这句话不填关系词可以吗？

不可以的，因为factory 是my grandfather worked的地点状语，所以必须加关系词。
意为我拜访了爷爷工作过的工厂。（我拜访了工厂---我拜访了哪个个工厂？---‘where’指出了是爷爷工作过的工厂）

I visited the factory where / in which my grandfather worked.
不可以省略.因为 work in the factory 是对的,work the factory 是错的.所以要用in which来替代in the factory,或者用where来替代.
(如果后面是 my grandfather worked in,则不用关系词可以)

区别一下:
I will play with the one you know.
the onw you know 是可以的,因为you know the one是对的.

不可以

要省略掉连词,只有当它是从句的主语时才可以.
如:
I find a flower (which) appears only in this area.

I don't want to make a friend with him (who) always lies to me.

当连词是作为句子的其他成分时,连词不能省略.

不可以，因为factory 做的是my grandfather worked的 地点状语。

不可以