yes, i would worry then. you have enough oil in the engine, right? if so, then your pump either has a clogged, or partially clogged pick up screen or its on is way out.
driving will result in damage to your bearings and rotating assemblies in the engien