Filtering is crucial in order to protect the packed column from all solid particles in feeds or eluents. Clogging of the column will increase back-pressure and disturb the feed loading zone causing reduced resolution and it is a common problem...
Everything needs to be filtered, every buffer every solution and solvent used, not only the feed solutions. To protect the packed column from solids always use in-line filters between the pump and the column. When an inlet frit becomes clogged it could result in a distorted flow profile, high back pressure in the column and suddenly, game over. So be sure to keep it clean by pre-filtering and regular replacement of the in-line filters.
If your feed is a bit too "cloggy" and it will take too long to filter, just do a quick centrifugation to get rid of the largest particles before starting the filtration. It will speed up the process.