The correct action would be to update your site's layout so that these files are outside your web server's document root -- then you don't have to worry about what to delete and what to leave. You probably have something like this, where your web server's document root is pointing directly at /path/to/project:

/path/to/project
    /lib
        foo.php
        bar.php
    /twig
        /twig
            /doc
            /test
    index.php

This means anybody can directly request http://yourdomain.com/twig/twig/test/some_file.php

What you want is more like this:

/path/to/project
    /public
        index.php
    /lib
        foo.php
        bar.php
    /twig
        /twig
            /doc
            /test

Then configure your web server so that its document root is /path/to/project/public. Then your application code can still include() things in /twig and /lib, but your web server won't directly serve them.

If your removing files from the vendor directory they'll come back the next time you do a composer install so this seems kind of pointless.

The files you mention (docs and test) are causing no harm other than taking up space as they are not directly called, but that's just a downside to any package management system. Like @Alex said as long as they are not publicly accessible there is no need to worry.

If you really want Twig without the extra files you could fork the project, move it into your own Git repo then reference that in your composer.json instead of the official one - but you will miss out on any updates from Twig.

We're running on Apache, so the easiest solution was to simply add a .htaccess file to the top vendor directory:

# Prevent non-local access to the vendor directory.
Order deny,allow
Deny from all

Directory structure:

/root
    /vendor
        .htaccess
        /twig
        ...
    /includes
        ...

Now the PHP scripts continue to have access, but external attempts to view anything inside the vendor directory return a 403 error.