GeoServer supports “vector tile” output in addition to the more standard image tile output. While the standard WMS output will generate a georeferenced map image, a vector tile contains georeferenced vector data, clipped into tiles for easy retrieval.
This is an output format, not a data source.
Here are two background video talks about “Vector Tiles with GeoServer and OpenLayers”:
Previous: Cross-layer filtering