Memory management

Not an easy topic. The problem is that both VTK and R use reference counting and the hard part is to synchronize both garbage collectors. The reference counter of VTK object is set to 1 when the object is created with its New method. The counter is incremented when object is inserted in other object, for example when the object is embedded in VTK processing pipeline. Delete method of vtkObjectBase class inherited by all VTK classes as well as removing object from processing pipeline decrements reference counter and memory deallocation takes place when counter drops to zero. The synchronization of VTK garbage collection is only one-way. If R owns a copy of object (i.e. the object have been created by explicit call to its New method inside R), then reg.finalizer function is applied to R's object instance. The reg.finalizer installs object's Delete method as a procedure to be called when the object no more referenced by R variables and garbage collection takes place.
Otherwise, the object can be created in the murky depth of VTK (we get a reference to object from call of VTK function) and the responsibility of memory deallocation in this case lays on VTK library itself. This is a bad situation and it may happen that referenced object is already deleted by VTK but we have a stale reference to it in R. Calling of methods of stale RVTK object will lead to segmentation fault and killing of R session. It would be highly desirable to install a notification callback which would clean up R environment upon object deletion, but I can not find an easy way to do it. For a while you can check the staleness of object by calling its GetReferenceCount() method. If return value is "unrealistic" (too big) then  the object reference is stale and you better should remove this object by rm() .
In example described above actor has reference count equal to 2. This is because it was set to 1 when we have created the actor (vtkActor()->actor) and it have been incremented when we add the actor to the pipeline (ren$AddActor(actor)). To clear the storage we have to remove the actor from renderer ( ren$RemoveActor(actor) ), overwrite ren variable ( ren<-NULL ) and call gc(). This procedure in safe way delete the object. Note that calling Delete method of R-object is in no way safe - the memory can be deallocated but object can be still needed by VTK pipeline, so it is wise not to touch Delete method. On other hand prop variable initially had reference count 1, because we have not created it ourselves, but have got a reference to it from actor$GetProperty() routine. When the actor is deleted the associated property is deleted too. Nonetheless the stale reference to prop variable is still looks like normal RVTK class. To check the RVTK class is stale you can call prop$GetReferenceCount() and then delete the variable by hand: rm("prop").