Especially with unified-shared memory and especially with C++'s virtual
functions, it is not uncommon to have on the device a function pointer
that points to the host function - but has an associated device.
If the pointed-to function is (explicitly or implicitly) 'declare target'
with the 'indirect' clause, it is added to the lookup table.
Before this commit, the conversion of the lookup table into a lookup
hash table happened every time a device kernel was launched on the first
team - albeit if already converted, the function immediately returned.
Ignoring the overhead, there was also a race: If multiple teams were
launched, it could happen that another team of the same target region
already tried to use the lookup table which it was still being created.
Likewise when lauching a kernel with 'nowait' and directly afterward
another kernel, there could be a race of creating the table.
With this commit, the creating of the kernel has been moved to the
host-plugin's GOMP_OFFLOAD_load_image. The previous code stored a
pointer to the host/device pointer array, which makes it hard when
creating the hash table on the host (data is needed for finding the
slot) - but accessing it on the device (where the lookup has to work
as well). As the hash-table implementation (only) supports integral
value as payload (0 and 1 having special meaning), the solution was
to move to an uint128_t variable to store both the host and device
address.
As the host-side library is typically dynamically linked and the
device-side one statically, there is the problem of backward
compatibility. The current implementation permits both older
binaries and newer libgomp and newer binaries with older libgomp.
I could imagine us breaking the latter eventually, but for now
there is up and downward compatibility. (Obviously, the race is
only fixed if new + new is combined.)
Code wise, on the device exist GOMP_INDIRECT_ADDR_MAP which was
updated to point to the host/device-address array. Now additionally
GOMP_INDIRECT_ADDR_HMAP exists, which contains the hash-table map.
If the latter exists, libgomp only updates it and the former remains
a NULL pointer; it is also untouched if there are no indirect functions.
Being NULL therefore avoids the call to the device-side build_indirect_map.
The code also currently supports to have no hash and a linear walk. I think
that remained from testing; due to the backward-compat feature, it can
actually be turned of on either side.
libgomp/ChangeLog:
PR libgomp/119857
PR libgomp/114445
* config/accel/target-indirect.c: Change to use uint128_t instead
of a struct as data structure and add GOMP_INDIRECT_ADDR_HMAP as
host-accessible variable.
(struct indirect_map_t): Remove.
(USE_HASHTAB_LOOKUP, INDIRECT_DEV_ADDR, INDIRECT_HOST_ADDR,
SET_INDIRECT_HOST_ADDR, SET_INDIRECT_ADDRS): Define.
(htab_free): Use __builtin_unreachable.
(htab_hash, htab_eq, GOMP_target_map_indirect_ptr,
build_indirect_map): Update for new representation and new
pointer-to-hash variable.
* config/gcn/team.c (gomp_gcn_enter_kernel): Only call
build_indirect_map when GOMP_INDIRECT_ADDR_MAP.
* config/nvptx/team.c (gomp_nvptx_main): Likewise.
* libgomp-plugin.h (GOMP_INDIRECT_ADDR_HMAP): Define.
* plugin/plugin-gcn.c: Conditionally include
build-target-indirect-htab.h.
(USE_HASHTAB_LOOKUP_FOR_INDIRECT): Define.
(create_target_indirect_map): New prototype.
(GOMP_OFFLOAD_load_image): Update to create the device's
indirect-function hash table on the host.
* plugin/plugin-nvptx.c: Conditionally include
build-target-indirect-htab.h.
(USE_HASHTAB_LOOKUP_FOR_INDIRECT): Define.
(create_target_indirect_map): New prototype.
(GOMP_OFFLOAD_load_image): Update to create the device's
indirect-function hash table on the host.
* plugin/build-target-indirect-htab.h: New file.
A splay-tree was previously used to lookup equivalent target addresses
for a given host address on offload targets. However, as splay-trees can
modify their structure on lookup, they are not suitable for concurrent
access from separate teams/threads without some form of locking. This
patch changes the lookup data structure to a hashtab instead, which does
not have these issues.
The call to build_indirect_map to initialize the data structure is now
called from just the first thread of the first team to avoid redundant
calls to this function.
2024-03-22 Kwok Cheung Yeung <kcyeung@baylibre.com>
libgomp/
* config/accel/target-indirect.c: Include string.h and hashtab.h.
Remove include of splay-tree.h. Update comments.
(splay_tree_prefix, splay_tree_c): Delete.
(struct indirect_map_t): New.
(hash_entry_type, htab_alloc, htab_free, htab_hash, htab_eq): New.
(GOMP_INDIRECT_ADD_MAP): Remove volatile qualifier.
(USE_SPLAY_TREE_LOOKUP): Rename to...
(USE_HASHTAB_LOOKUP): ..this.
(indirect_map, indirect_array): Delete.
(indirect_htab): New.
(build_indirect_map): Remove locking. Build indirect map using
hashtab.
(GOMP_target_map_indirect_ptr): Use indirect_htab to lookup target
address.
(GOMP_target_map_indirect_ptr): Remove volatile qualifier.
* config/gcn/team.c (gomp_gcn_enter_kernel): Call build_indirect_map
from first thread of first team only.
* config/nvptx/team.c (gomp_nvptx_main): Likewise.
* testsuite/libgomp.c-c++-common/declare-target-indirect-2.c (main):
Add missing break statements.
* testsuite/libgomp.fortran/declare-target-indirect-2.f90: Remove
xfail.
This adds support for the 'indirect' clause in the 'declare target'
directive. Functions declared as indirect may be called via function
pointers passed from the host in offloaded code.
Virtual calls to member functions via the object pointer in C++ are
currently not supported in target regions.
2023-11-07 Kwok Cheung Yeung <kcy@codesourcery.com>
gcc/c-family/
* c-attribs.cc (c_common_attribute_table): Add attribute for
indirect functions.
* c-pragma.h (enum parma_omp_clause): Add entry for indirect clause.
gcc/c/
* c-decl.cc (c_decl_attributes): Add attribute for indirect
functions.
* c-lang.h (c_omp_declare_target_attr): Add indirect field.
* c-parser.cc (c_parser_omp_clause_name): Handle indirect clause.
(c_parser_omp_clause_indirect): New.
(c_parser_omp_all_clauses): Handle indirect clause.
(OMP_DECLARE_TARGET_CLAUSE_MASK): Add indirect clause to mask.
(c_parser_omp_declare_target): Handle indirect clause. Emit error
message if device_type or indirect clauses used alone. Emit error
if indirect clause used with device_type that is not 'any'.
(OMP_BEGIN_DECLARE_TARGET_CLAUSE_MASK): Add indirect clause to mask.
(c_parser_omp_begin): Handle indirect clause.
* c-typeck.cc (c_finish_omp_clauses): Handle indirect clause.
gcc/cp/
* cp-tree.h (cp_omp_declare_target_attr): Add indirect field.
* decl2.cc (cplus_decl_attributes): Add attribute for indirect
functions.
* parser.cc (cp_parser_omp_clause_name): Handle indirect clause.
(cp_parser_omp_clause_indirect): New.
(cp_parser_omp_all_clauses): Handle indirect clause.
(handle_omp_declare_target_clause): Add extra parameter. Add
indirect attribute for indirect functions.
(OMP_DECLARE_TARGET_CLAUSE_MASK): Add indirect clause to mask.
(cp_parser_omp_declare_target): Handle indirect clause. Emit error
message if device_type or indirect clauses used alone. Emit error
if indirect clause used with device_type that is not 'any'.
(OMP_BEGIN_DECLARE_TARGET_CLAUSE_MASK): Add indirect clause to mask.
(cp_parser_omp_begin): Handle indirect clause.
* semantics.cc (finish_omp_clauses): Handle indirect clause.
gcc/
* lto-cgraph.cc (enum LTO_symtab_tags): Add tag for indirect
functions.
(output_offload_tables): Write indirect functions.
(input_offload_tables): read indirect functions.
* lto-section-names.h (OFFLOAD_IND_FUNC_TABLE_SECTION_NAME): New.
* omp-builtins.def (BUILT_IN_GOMP_TARGET_MAP_INDIRECT_PTR): New.
* omp-offload.cc (offload_ind_funcs): New.
(omp_discover_implicit_declare_target): Add functions marked with
'omp declare target indirect' to indirect functions list.
(omp_finish_file): Add indirect functions to section for offload
indirect functions.
(execute_omp_device_lower): Redirect indirect calls on target by
passing function pointer to BUILT_IN_GOMP_TARGET_MAP_INDIRECT_PTR.
(pass_omp_device_lower::gate): Run pass_omp_device_lower if
indirect functions are present on an accelerator device.
* omp-offload.h (offload_ind_funcs): New.
* tree-core.h (omp_clause_code): Add OMP_CLAUSE_INDIRECT.
* tree.cc (omp_clause_num_ops): Add entry for OMP_CLAUSE_INDIRECT.
(omp_clause_code_name): Likewise.
* tree.h (OMP_CLAUSE_INDIRECT_EXPR): New.
* config/gcn/mkoffload.cc (process_asm): Process offload_ind_funcs
section. Count number of indirect functions.
(process_obj): Emit number of indirect functions.
* config/nvptx/mkoffload.cc (ind_func_ids, ind_funcs_tail): New.
(process): Emit offload_ind_func_table in PTX code. Emit indirect
function names and count in image.
* config/nvptx/nvptx.cc (nvptx_record_offload_symbol): Mark
indirect functions in PTX code with IND_FUNC_MAP.
gcc/testsuite/
* c-c++-common/gomp/declare-target-7.c: Update expected error message.
* c-c++-common/gomp/declare-target-indirect-1.c: New.
* c-c++-common/gomp/declare-target-indirect-2.c: New.
* g++.dg/gomp/attrs-21.C (v12): Update expected error message.
* g++.dg/gomp/declare-target-indirect-1.C: New.
* gcc.dg/gomp/attrs-21.c (v12): Update expected error message.
include/
* gomp-constants.h (GOMP_VERSION): Increment to 3.
(GOMP_VERSION_SUPPORTS_INDIRECT_FUNCS): New.
libgcc/
* offloadstuff.c (OFFLOAD_IND_FUNC_TABLE_SECTION_NAME): New.
(__offload_ind_func_table): New.
(__offload_ind_funcs_end): New.
(__OFFLOAD_TABLE__): Add entries for indirect functions.
libgomp/
* Makefile.am (libgomp_la_SOURCES): Add target-indirect.c.
* Makefile.in: Regenerate.
* libgomp-plugin.h (GOMP_INDIRECT_ADDR_MAP): New define.
(GOMP_OFFLOAD_load_image): Add extra argument.
* libgomp.h (struct indirect_splay_tree_key_s): New.
(indirect_splay_tree_node, indirect_splay_tree,
indirect_splay_tree_key): New.
(indirect_splay_compare): New.
* libgomp.map (GOMP_5.1.1): Add GOMP_target_map_indirect_ptr.
* libgomp.texi (OpenMP 5.1): Update documentation on indirect
calls in target region and on indirect clause.
(Other new OpenMP 5.2 features): Add entry for virtual function calls.
* libgomp_g.h (GOMP_target_map_indirect_ptr): Add prototype.
* oacc-host.c (host_load_image): Add extra argument.
* target.c (gomp_load_image_to_device): If the GOMP_VERSION is high
enough, read host indirect functions table and pass to
load_image_func.
* config/accel/target-indirect.c: New.
* config/linux/target-indirect.c: New.
* config/gcn/team.c (build_indirect_map): Add prototype.
(gomp_gcn_enter_kernel): Initialize support for indirect
function calls on GCN target.
* config/nvptx/team.c (build_indirect_map): Add prototype.
(gomp_nvptx_main): Initialize support for indirect function
calls on NVPTX target.
* plugin/plugin-gcn.c (struct gcn_image_desc): Add field for
indirect functions count.
(GOMP_OFFLOAD_load_image): Add extra argument. If the GOMP_VERSION
is high enough, build address translation table and copy it to target
memory.
* plugin/plugin-nvptx.c (nvptx_tdata): Add field for indirect
functions count.
(GOMP_OFFLOAD_load_image): Add extra argument. If the GOMP_VERSION
is high enough, Build address translation table and copy it to target
memory.
* testsuite/libgomp.c-c++-common/declare-target-indirect-1.c: New.
* testsuite/libgomp.c-c++-common/declare-target-indirect-2.c: New.
* testsuite/libgomp.c++/declare-target-indirect-1.C: New.
This patch introduces gomp_sem_getcount wrapper, which uses sem_getvalue
for POSIX and atomic loads for linux futex and accel. rtems for now
remains broken.
2021-01-18 Jakub Jelinek <jakub@redhat.com>
* config/linux/sem.h (gomp_sem_getcount): New function.
* config/posix/sem.h (gomp_sem_getcount): New function.
* config/posix/sem.c (gomp_sem_getcount): New function.
* config/accel/sem.h (gomp_sem_getcount): New function.
* task.c (task_fulfilled_p): Use gomp_sem_getcount.
(omp_fulfill_event): Likewise.
There's no point in using value '-3', and even though not directly related,
value '-1' does match 'GOMP_DEVICE_ICV'.
libgomp/
* config/accel/openacc.f90 (acc_device_current): Set to '-1'.
* openacc.f90 (acc_device_current): Likewise.
* openacc.h (acc_device_current): Likewise.
* openacc_lib.h (acc_device_current): Likewise.
2020-02-19 Tobias Burnus <tobias@codesourcery.com>
* .gitattributes: New; whitespace handling for Fortran's openacc_lib.h.
* config/accel/openacc.f90 (openacc_kinds): Add acc_device_current.
(openacc_internal, acc_on_device_h): Fix argument name; minor cleanup.
* libgomp.texi (Enabling OpenACC): No longer mark as experimental.
(acc_set_device_num): Fix Fortran argument name, use same name for C.
(acc_get_property): Update Fortran interface to post-OpenACC 3.0
corrections; add note about the previous interface and named constant.
(OpenACC library and environment variables): Fix two typos.
* openacc.f90: Use for all procedures the argument names from the spec
as for …_h they are user visible.
(openacc_kinds): Rename acc_device_property to
acc_device_property_kinds and change value to int32 ; and update users.
Re-add acc_device_property for for backward compatibility.
(acc_get_property_string_h): Clean up as acc_device_property_kind
changed.
(acc_get_property_h): Likewise and return c_size_t instead of
acc_device_property.
(openacc): Also export acc_device_property_kinds.
(acc_async_test_h, acc_async_test_all_h, acc_on_device_h,
acc_is_present_32_h, acc_is_present_64_h): Simplify logical-return-value
handling; check against /= 0 instead of == 1 to match C.
* openacc_lib.h: Use for all procedures the argument names from the spec
as for …_h they are user visible. Place !GCC$ into the first column to
be active also for fixed-form souce form.
(acc_device_current, acc_device_property_kind, acc_device_property,
acc_property_memory, acc_property_free_memory, acc_property_name,
acc_property_vendor, acc_property_driver): New named constants.
(acc_get_property, acc_get_property_string): New generic interface.
* config/accel/openacc.f90 (module openacc_kinds): Use 'PUBLIC' to mark
all symbols as public except for the 'use …, only' imported symbol,
which is private.
(module openacc): Default to 'PRIVATE' to exclude openacc_internal; mark
all symbols from module openacc_kinds as PUBLIC
* openacc.f90: Add comment with crossref to that file and openmp_lib.h;
fix comment typo.
* openacc_lib.h (acc_device_gcn): Add this PARAMETER.
From-SVN: r279456
