It looks like I only read half the resolution of LWG 3480 and decided we
already supported it. As well as making the non-member overloads of end
take their parameters by value, we need some specializations of the
enable_borrowed_range and enable_view variable templates.
libstdc++-v3/ChangeLog:
PR libstdc++/117560
* include/bits/fs_dir.h (enable_borrowed_range, enable_view):
Define specializations for directory iterators, as per LWG 3480.
* testsuite/27_io/filesystem/iterators/lwg3480.cc: New test.
This library defect was recently approved for C++23.
libstdc++-v3/ChangeLog:
* include/bits/fs_dir.h (directory_iterator): Add comparison
with std::default_sentinel_t. Remove redundant operator!= for
C++20.
* (recursive_directory_iterator): Likewise.
* include/bits/iterator_concepts.h [!__cpp_lib_concepts]
(default_sentinel_t, default_sentinel): Define even if concepts
are not supported.
* include/bits/regex.h (regex_iterator): Add comparison with
std::default_sentinel_t. Remove redundant operator!= for C++20.
(regex_token_iterator): Likewise.
(regex_token_iterator::_M_end_of_seq()): Add noexcept.
* testsuite/27_io/filesystem/iterators/lwg3719.cc: New test.
* testsuite/28_regex/iterators/regex_iterator/lwg3719.cc:
New test.
* testsuite/28_regex/iterators/regex_token_iterator/lwg3719.cc:
New test.
This fixes the remaining filesystem::remove_all race condition by using
POSIX openat to recurse into sub-directories and using POSIX unlinkat to
remove files. This avoids the remaining race where the directory being
removed is replaced with a symlink after the directory has been opened,
so that the filesystem::remove("subdir/file") resolves to "target/file"
instead, because "subdir" has been removed and replaced with a symlink.
The previous patch only fixed the case where the directory was replaced
with a symlink before we tried to open it, but it still used the full
(potentially compromised) path as an argument to filesystem::remove.
The first part of the fix is to use openat when recursing into a
sub-directory with recursive_directory_iterator. This means that opening
"dir/subdir" uses the file descriptor for "dir", and so is sure to open
"dir/subdir" and not "symlink/subdir". (The previous patch to use
O_NOFOLLOW already ensured we won't open "dir/symlink/" here.)
The second part of the fix is to use unlinkat for the remove_all
operation. Previously we used a directory_iterator to get the name of
each file in a directory and then used filesystem::remove(iter->path())
on that name. This meant that any checks (e.g. O_NOFOLLOW) done by the
iterator could be invalidated before the remove operation on that
pathname. The directory iterator contains an open DIR stream, which we
can use to obtain a file descriptor to pass to unlinkat. This ensures
that the file being deleted really is contained within the directory
we're iterating over, rather than using a pathname that could resolve to
some other file.
The filesystem::remove_all function previously used a (non-recursive)
filesystem::directory_iterator for each directory, and called itself
recursively for sub-directories. The new implementation uses a single
filesystem::recursive_directory_iterator object, and calls a new __erase
member function on that iterator. That new __erase member function does
the actual work of removing a file (or a directory after its contents
have been iterated over and removed) using unlinkat. That means we don't
need to expose the DIR stream or its file descriptor to the remove_all
function, it's still encapuslated by the iterator class.
It would be possible to add a __rewind member to directory iterators
too, to call rewinddir after each modification to the directory. That
would make it more likely for filesystem::remove_all to successfully
remove everything even if files are being written to the directory tree
while removing it. It's unclear if that is actually prefereable, or if
it's better to fail and report an error at the first opportunity.
The necessary APIs (openat, unlinkat, fdopendir, dirfd) are defined in
POSIX.1-2008, and in Glibc since 2.10. But if the target doesn't provide
them, the original code (with race conditions) is still used.
This also reduces the number of small memory allocations needed for
std::filesystem::remove_all, because we do not store the full path to
every directory entry that is iterated over. The new filename_only
option means we only store the filename in the directory entry, as that
is all we need in order to use openat or unlinkat.
Finally, rather than duplicating everything for the Filesystem TS, the
std::experimental::filesystem::remove_all implementation now just calls
std::filesystem::remove_all to do the work.
libstdc++-v3/ChangeLog:
PR libstdc++/104161
* acinclude.m4 (GLIBCXX_CHECK_FILESYSTEM_DEPS): Check for dirfd
and unlinkat.
* config.h.in: Regenerate.
* configure: Regenerate.
* include/bits/fs_dir.h (recursive_directory_iterator): Declare
remove_all overloads as friends.
(recursive_directory_iterator::__erase): Declare new member
function.
* include/bits/fs_fwd.h (remove, remove_all): Declare.
* src/c++17/fs_dir.cc (_Dir): Add filename_only parameter to
constructor. Pass file descriptor argument to base constructor.
(_Dir::dir_and_pathname, _Dir::open_subdir, _Dir::do_unlink)
(_Dir::unlink, _Dir::rmdir): Define new member functions.
(directory_iterator): Pass filename_only argument to _Dir
constructor.
(recursive_directory_iterator::_Dir_stack): Adjust constructor
parameters to take a _Dir rvalue instead of creating one.
(_Dir_stack::orig): Add data member for storing original path.
(_Dir_stack::report_error): Define new member function.
(__directory_iterator_nofollow): Move here from dir-common.h and
fix value to be a power of two.
(__directory_iterator_filename_only): Define new constant.
(recursive_directory_iterator): Construct _Dir object and move
into _M_dirs stack. Pass skip_permission_denied argument to first
advance call.
(recursive_directory_iterator::increment): Use _Dir::open_subdir.
(recursive_directory_iterator::__erase): Define new member
function.
* src/c++17/fs_ops.cc (ErrorReporter, do_remove_all): Remove.
(fs::remove_all): Use new recursive_directory_iterator::__erase
member function.
* src/filesystem/dir-common.h (_Dir_base): Add int parameter to
constructor and use openat to implement nofollow semantics.
(_Dir_base::fdcwd, _Dir_base::set_close_on_exec, _Dir_base::openat):
Define new member functions.
(__directory_iterator_nofollow): Move to fs_dir.cc.
* src/filesystem/dir.cc (_Dir): Pass file descriptor argument to
base constructor.
(_Dir::dir_and_pathname, _Dir::open_subdir): Define new member
functions.
(recursive_directory_iterator::_Dir_stack): Adjust constructor
parameters to take a _Dir rvalue instead of creating one.
(recursive_directory_iterator): Check for new nofollow option.
Construct _Dir object and move into _M_dirs stack. Pass
skip_permission_denied argument to first advance call.
(recursive_directory_iterator::increment): Use _Dir::open_subdir.
* src/filesystem/ops.cc (fs::remove_all): Use C++17 remove_all.
The equality operators for directory iterators are not explicitly
specified in the standard, they're only required to meet the iterator
requirements. This means we don't need to declare them at namespace
scope and can implement them as hidden friends.
Also add 'noexcept' to directory_iterator's dereference operators.
* include/bits/fs_dir.h (directory_iterator::operator*)
(directory_iterator::operator->): Add noexcept.
(operator==, operator!=): Replace namespace-scope equality operators
for directory iterators with hidden friends.
From-SVN: r270175
Implement the proposed resolution of LWG 2708 by moving the _M_options
and _M_pending members out of the recursive_directory_iterator into the
shared _Dir_stack object. Because _Dir_stack is an opaque type, the
member functions that access the _M_options and _M_pending variables
cannot be inline. Move them into the library.
As a drive-by fix, add noexcept to the non-throwing member functions of
recursive_directory_iterator.
* config/abi/pre/gnu.ver: Export new symbols.
* include/bits/fs_dir.h (recursive_directory_iterator::options())
(recursive_directory_iterator::recursion_pending())
(recursive_directory_iterator::disable_recursion_pending()): Remove
inline definitions. Make noexcept.
(recursive_directory_iterator::depth())
(recursive_directory_iterator::operator*())
(recursive_directory_iterator::operator->()): Make noexcept.
(recursive_directory_iterator::_M_options)
(recursive_directory_iterator::_M_pending): Remove data members.
* src/c++17/fs_path.cc (_Dir_stack): Add constructor and data members.
(recursive_directory_iterator::recursive_directory_iterator): Remove
ctor-initializer. Use new constructor for _Dir_stack.
(recursive_directory_iterator::options())
(recursive_directory_iterator::recursion_pending())
(recursive_directory_iterator::disable_recursion_pending()): Add
non-inline definitions.
(recursive_directory_iterator::depth()): Make noexcept.
(recursive_directory_iterator::increment(error_code&))
(recursive_directory_iterator::pop(error_code&)): Adjust to new
location of options and recursion_pending members.
* testsuite/27_io/filesystem/iterators/recursion_pending.cc: New test.
* testsuite/util/testsuite_fs.h (__gnu_test::scoped_file): Add
user-declared move constructor and assignment operator, to make the
type move-only.
From-SVN: r270173
* include/bits/fs_dir.h (operator<<): Overload for directory_entry,
as per LWG 3171.
* testsuite/27_io/filesystem/directory_entry/lwg3171.cc: New test.
From-SVN: r267238
This resolves a longstanding issue where the lock policy for shared_ptr
reference counting depends on compilation options when the header is
included, so that different -march options can cause ABI changes. For
example, objects compiled with -march=armv7 will use atomics to
synchronize reference counts, and objects compiled with -march=armv5t
will use a mutex. That means the shared_ptr control block will have a
different layout in different objects, causing ODR violations and
undefined behaviour. This was the root cause of PR libstdc++/42734 as
well as PR libstdc++/67843.
The solution is to decide on the lock policy at build time, when
libstdc++ is configured. The configure script checks for the
availability of the necessary atomic built-ins for the target and fixes
that choice permanently. Different -march flags used to compile user
code will not cause changes to the lock policy. This results in an ABI
change for certain compilations, but only where there was already an ABI
incompatibility between the libstdc++.so library and objects built with
an incompatible -march option. In general, this means a more stable ABI
that isn't silently altered when -march flags make addition atomic ops
available.
To force a target to use "atomic" or "mutex" the new configure option
--with-libstdcxx-lock-policy can be used.
In order to turn ODR violations into linker errors, the uses of
shared_ptr in filesystem directory iterators have been replaced
with __shared_ptr, and explicit instantiations are declared. This
ensures that object files using those types cannot link to libstdc++
libs unless they use the same lock policy.
PR libstdc++/67843
* acinclude.m4 (GLIBCXX_ENABLE_LOCK_POLICY): Add new macro
that defines _GLIBCXX_HAVE_ATOMIC_LOCK_POLICY.
* config.h.in: Regenerate.
* configure: Regenerate.
* configure.ac: Use GLIBCXX_ENABLE_LOCK_POLICY.
* doc/xml/manual/configure.xml: Document new configure option.
* include/bits/fs_dir.h (directory_iterator): Use __shared_ptr
instead of shared_ptr.
(recursive_directory_iterator): Likewise.
(__shared_ptr<_Dir>): Add explicit instantiation declaration.
(__shared_ptr<recursive_directory_iterator::_Dir_stack>): Likewise.
* include/bits/shared_ptr_base.h (__allocate_shared, __make_shared):
Add default template argument for _Lock_policy template parameter.
* include/ext/concurrence.h (__default_lock_policy): Check macro
_GLIBCXX_HAVE_ATOMIC_LOCK_POLICY instead of checking if the current
target supports the builtins for compare-and-swap.
* src/filesystem/std-dir.cc (__shared_ptr<_Dir>): Add explicit
instantiation definition.
(__shared_ptr<recursive_directory_iterator::_Dir_stack>): Likewise.
(directory_iterator, recursive_directory_iterator): Use __make_shared
instead of make_shared.
From-SVN: r266533
PR libstdc++/86597
* include/bits/fs_dir.h (directory_entry::_M_file_type(error_code&)):
Clear error_code when cached type is used.
* testsuite/27_io/filesystem/directory_entry/86597.cc: New test.
From-SVN: r263397
Based on Filesystem TS implementation, with the changes applied by:
- P0219R1 Relative Paths for Filesystem
- P0317R1 Directory Entry Caching for Filesystem
- P0492R2 Resolution of C++17 National Body Comments
Where appropriate code is shared between the TS and C++17
implementations.
* include/Makefile.am: Add new headers for C++17 filesystem library.
* include/Makefile.in: Regenerate.
* include/bits/fs_dir.h: New header, based on Filesystem TS code in
include/experimental/bits directory.
* include/bits/fs_fwd.h: Likewise.
* include/bits/fs_ops.h: Likewise.
* include/bits/fs_path.h: Likewise.
* include/experimental/bits/fs_dir.h: Rename Doxygen group.
* include/experimental/bits/fs_fwd.h: Likewise.
* include/experimental/bits/fs_ops.h: Likewise.
* include/experimental/bits/fs_path.h: Likewise.
* include/experimental/filesystem (filesystem_error::_M_gen_what):
Remove inline definition.
* include/precompiled/stdc++.h: Add <filesystem> to precompiled
header.
* include/std/filesystem: New header.
* python/libstdcxx/v6/printers.py: Enable printer for std::filesystem
paths.
* src/filesystem/Makefile.am: Add new files. Compile as C++17.
* src/filesystem/Makefile.in: Regenerate.
* src/filesystem/cow-dir.cc: Update comment.
* src/filesystem/cow-ops.cc: Likewise.
* src/filesystem/cow-path.cc: Likewise.
* src/filesystem/cow-std-dir.cc: New file.
* src/filesystem/cow-std-ops.cc: New file.
* src/filesystem/cow-std-path.cc: New file.
* src/filesystem/dir-common.h (_Dir_base, get_file_type): New header
for common code.
* src/filesystem/dir.cc (_Dir): Derive from _Dir_base.
(open_dir): Move to _Dir_base constructor.
(get_file_type): Move to dir-common.h.
(recurse): Move to _Dir_base::should_recurse.
* src/filesystem/ops-common.h: New header for common code.
* src/filesystem/ops.cc (is_set, make_file_type, make_file_status)
(is_not_found_errno, file_time, do_copy_file): Move to ops-common.h.
* src/filesystem/path.cc (filesystem_error::_M_gen_what): Define.
* src/filesystem/std-dir.cc: New file, based on Filesystem TS code.
* src/filesystem/std-ops.cc: Likewise.
* src/filesystem/std-dir.cc: Likewise.
* testsuite/27_io/filesystem/iterators/directory_iterator.cc: New
test.
* testsuite/27_io/filesystem/iterators/pop.cc: New test.
* testsuite/27_io/filesystem/iterators/recursive_directory_iterator.cc:
New test.
* testsuite/27_io/filesystem/operations/absolute.cc: New test.
* testsuite/27_io/filesystem/operations/canonical.cc: New test.
* testsuite/27_io/filesystem/operations/copy.cc: New test.
* testsuite/27_io/filesystem/operations/copy_file.cc: New test.
* testsuite/27_io/filesystem/operations/create_directories.cc: New
test.
* testsuite/27_io/filesystem/operations/create_directory.cc: New test.
* testsuite/27_io/filesystem/operations/create_symlink.cc: New test.
* testsuite/27_io/filesystem/operations/current_path.cc: New test.
* testsuite/27_io/filesystem/operations/equivalent.cc: New test.
* testsuite/27_io/filesystem/operations/exists.cc: New test.
* testsuite/27_io/filesystem/operations/file_size.cc: New test.
* testsuite/27_io/filesystem/operations/is_empty.cc: New test.
* testsuite/27_io/filesystem/operations/last_write_time.cc: New test.
* testsuite/27_io/filesystem/operations/permissions.cc: New test.
* testsuite/27_io/filesystem/operations/proximate.cc: New test.
* testsuite/27_io/filesystem/operations/read_symlink.cc: New test.
* testsuite/27_io/filesystem/operations/relative.cc: New test.
* testsuite/27_io/filesystem/operations/remove_all.cc: New test.
* testsuite/27_io/filesystem/operations/space.cc: New test.
* testsuite/27_io/filesystem/operations/status.cc: New test.
* testsuite/27_io/filesystem/operations/symlink_status.cc: New test.
* testsuite/27_io/filesystem/operations/temp_directory_path.cc: New
test.
* testsuite/27_io/filesystem/operations/weakly_canonical.cc: New test.
* testsuite/27_io/filesystem/path/append/path.cc: New test.
* testsuite/27_io/filesystem/path/assign/assign.cc: New test.
* testsuite/27_io/filesystem/path/assign/copy.cc: New test.
* testsuite/27_io/filesystem/path/compare/compare.cc: New test.
* testsuite/27_io/filesystem/path/compare/path.cc: New test.
* testsuite/27_io/filesystem/path/compare/strings.cc: New test.
* testsuite/27_io/filesystem/path/concat/path.cc: New test.
* testsuite/27_io/filesystem/path/concat/strings.cc: New test.
* testsuite/27_io/filesystem/path/construct/copy.cc: New test.
* testsuite/27_io/filesystem/path/construct/default.cc: New test.
* testsuite/27_io/filesystem/path/construct/locale.cc: New test.
* testsuite/27_io/filesystem/path/construct/range.cc: New test.
* testsuite/27_io/filesystem/path/construct/string_view.cc: New test.
* testsuite/27_io/filesystem/path/decompose/extension.cc: New test.
* testsuite/27_io/filesystem/path/decompose/filename.cc: New test.
* testsuite/27_io/filesystem/path/decompose/parent_path.cc: New test.
* testsuite/27_io/filesystem/path/decompose/relative_path.cc: New
test.
* testsuite/27_io/filesystem/path/decompose/root_directory.cc: New
test.
* testsuite/27_io/filesystem/path/decompose/root_name.cc: New test.
* testsuite/27_io/filesystem/path/decompose/root_path.cc: New test.
* testsuite/27_io/filesystem/path/decompose/stem.cc: New test.
* testsuite/27_io/filesystem/path/generation/normal.cc: New test.
* testsuite/27_io/filesystem/path/generation/proximate.cc: New test.
* testsuite/27_io/filesystem/path/generation/relative.cc: New test.
* testsuite/27_io/filesystem/path/generic/generic_string.cc: New test.
* testsuite/27_io/filesystem/path/itr/traversal.cc: New test.
* testsuite/27_io/filesystem/path/modifiers/clear.cc: New test.
* testsuite/27_io/filesystem/path/modifiers/make_preferred.cc: New
test.
* testsuite/27_io/filesystem/path/modifiers/remove_filename.cc: New
test.
* testsuite/27_io/filesystem/path/modifiers/replace_extension.cc: New
test.
* testsuite/27_io/filesystem/path/modifiers/replace_filename.cc: New
test.
* testsuite/27_io/filesystem/path/modifiers/swap.cc: New test.
* testsuite/27_io/filesystem/path/native/string.cc: New test.
* testsuite/27_io/filesystem/path/nonmember/hash_value.cc: New test.
* testsuite/27_io/filesystem/path/query/empty.cc: New test.
* testsuite/27_io/filesystem/path/query/has_extension.cc: New test.
* testsuite/27_io/filesystem/path/query/has_filename.cc: New test.
* testsuite/27_io/filesystem/path/query/has_parent_path.cc: New test.
* testsuite/27_io/filesystem/path/query/has_relative_path.cc: New
test.
* testsuite/27_io/filesystem/path/query/has_root_directory.cc: New
test.
* testsuite/27_io/filesystem/path/query/has_root_name.cc: New test.
* testsuite/27_io/filesystem/path/query/has_root_path.cc: New test.
* testsuite/27_io/filesystem/path/query/has_stem.cc: New test.
* testsuite/27_io/filesystem/path/query/is_relative.cc: New test.
* testsuite/experimental/filesystem/path/construct/string_view.cc:
Define USE_FILESYSTEM_TS.
* testsuite/util/testsuite_fs.h: Allow use with C++17 paths as well
as Filesystem TS.
From-SVN: r254008
