yes,  i would worry then.  you have enough oil in the engine,  right?  if so,  then your pump either has a clogged, or partially clogged pick up screen or its on is way out.
driving will result in damage to your bearings and rotating assemblies in the engien