If your file system is badly fragmented, and the file in question was comprised of many fragments (also sometimes called "extents"), then you could see a file deletion take a bit of time.

When you delete a file, it is true that (unless otherwise configured), no data is overwritten and there are only pointers being pulled off one list and put on another. However, if you have a badly fragmented file, there isn't just one pointer being moved for one file -- there is the file header, and then the header for every non-contiguous extension of the file's storage area that is being pulled off one list and being put on another.

This is one of the reasons why de-fragmenting a file system can speed disk operations up.

The inverse is true, too. If you're copying a file, and your filesystem is badly fragmented, the creation of the new file has to be done in many small pieces and linked together, rather than one creation of the size you need.