pax_global_header00006660000000000000000000000064150011307240014503gustar00rootroot0000000000000052 comment=77201cb9c00f140422db88decc109a8efbe8eba6 ora2pg-25.0/000077500000000000000000000000001500113072400126235ustar00rootroot00000000000000ora2pg-25.0/INSTALL000066400000000000000000000000251500113072400136510ustar00rootroot00000000000000 See README file. ora2pg-25.0/LICENSE000066400000000000000000000773301500113072400136420ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS ora2pg-25.0/MANIFEST000066400000000000000000000002641500113072400137560ustar00rootroot00000000000000changelog INSTALL README LICENSE MANIFEST Makefile.PL lib/Ora2Pg.pm lib/Ora2Pg/PLSQL.pm lib/Ora2Pg/GEOM.pm lib/Ora2Pg/MySQL.pm scripts/ora2pg scripts/ora2pg_scanner doc/Ora2Pg.pod ora2pg-25.0/Makefile.PL000066400000000000000000002251131500113072400146010ustar00rootroot00000000000000use ExtUtils::MakeMaker qw(prompt WriteMakefile); my @ALLOWED_ARGS = ('CONFDIR','DOCDIR','DESTDIR','QUIET','INSTALLDIRS','INSTALL_BASE','PREFIX'); # Parse command line arguments and store them as environment variables while ($_ = shift) { my ($k,$v) = split(/=/, $_, 2); if (grep(/^$k$/, @ALLOWED_ARGS)) { $ENV{$k} = $v; } } # Default install path my $CONFDIR = $ENV{CONFDIR} || '/etc/ora2pg'; my $RPM_CONFDIR = $CONFDIR; my $DOCDIR = $ENV{DOCDIR} || '/usr/local/share/doc/ora2pg'; my $DEST_CONF_FILE = 'ora2pg.conf.dist'; my $DATA_LIMIT_DEFAULT = 10000; if ($^O =~ /MSWin32|dos/i) { $DATA_LIMIT_DEFAULT = 2000; } my $PREFIX = $ENV{DESTDIR} || $ENV{PREFIX} || $ENV{INSTALL_BASE} || ''; $PREFIX =~ s/\/$//; $ENV{INSTALLDIRS} ||= 'site'; # Try to set the default configuration directory following $PREFIX if ($^O =~ /MSWin32|dos/i) { # Force default path $CONFDIR = 'C:\ora2pg'; $DOCDIR = 'C:\ora2pg'; } elsif ($PREFIX) { if (!$ENV{CONFDIR}) { $CONFDIR = $PREFIX . '/etc/ora2pg'; } else { $CONFDIR = $PREFIX . '/' . $ENV{CONFDIR}; } if (!$ENV{DOCDIR}) { $DOCDIR = $PREFIX . '/doc/ora2pg'; } else { $DOCDIR = $PREFIX . '/' . $ENV{DOCDIR}; } } # Try to find all binary used by Ora2Pg my $bzip2 = ''; if ($^O !~ /MSWin32|dos/i) { my $bzip2 = `which bzip2`; chomp($bzip2); $bzip2 ||= '/usr/bin/bzip2'; } my $oracle_home = $ENV{ORACLE_HOME} || '/usr/local/oracle/10g'; # Setup ok. generating default ora2pg.conf config file unless(open(OUTCFG, ">$DEST_CONF_FILE")) { print "\nError: can't write config file $DEST_CONF_FILE, $!\n"; exit 0; } print OUTCFG qq{ #################### Ora2Pg Configuration file ##################### # Support for including a common config file that may contain any # of the following configuration directives. #IMPORT common.conf #------------------------------------------------------------------------------ # INPUT SECTION (Oracle connection or input file) #------------------------------------------------------------------------------ # Set this directive to a file containing PL/SQL Oracle Code like function, # procedure or a full package body to prevent Ora2Pg from connecting to an # Oracle database end just apply his conversion tool to the content of the # file. This can only be used with the following export type: PROCEDURE, # FUNCTION or PACKAGE. If you don't know what you do don't use this directive. #INPUT_FILE ora_plsql_src.sql # Set the Oracle home directory ORACLE_HOME $oracle_home # Set Oracle database connection (datasource, user, password) ORACLE_DSN dbi:Oracle:host=mydb.mydom.fr;sid=SIDNAME;port=1521 ORACLE_USER system ORACLE_PWD manager # Set this to 1 if you connect as simple user and can not extract things # from the DBA_... tables. It will use tables ALL_... This will not works # with GRANT export, you should use an Oracle DBA username at ORACLE_USER USER_GRANTS 0 # Trace all to stderr DEBUG 0 # This directive can be used to send an initial command to Oracle, just after # the connection. For example to unlock a policy before reading objects or # to set some session parameters. This directive can be used multiple time. #ORA_INITIAL_COMMAND #------------------------------------------------------------------------------ # SCHEMA SECTION (Oracle schema to export and use of schema in PostgreSQL) #------------------------------------------------------------------------------ # Export Oracle schema to PostgreSQL schema EXPORT_SCHEMA 0 # Oracle schema/owner to use #SCHEMA SCHEMA_NAME # Enable/disable the CREATE SCHEMA SQL order at starting of the output file. # It is enable by default and concern on TABLE export type. CREATE_SCHEMA 1 # Enable this directive to force Oracle to compile schema before exporting code. # When this directive is enabled and SCHEMA is set to a specific schema name, # only invalid objects in this schema will be recompiled. If SCHEMA is not set # then all schema will be recompiled. To force recompile invalid object in a # specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. # This will ask to Oracle to validate the PL/SQL that could have been invalidate # after a export/import for example. The 'VALID' or 'INVALID' status applies to # functions, procedures, packages and user defined types. COMPILE_SCHEMA 1 # By default if you set EXPORT_SCHEMA to 1 the PostgreSQL search_path will be # set to the schema name exported set as value of the SCHEMA directive. You can # defined/force the PostgreSQL schema to use by using this directive. # # The value can be a comma delimited list of schema but not when using TABLE # export type because in this case it will generate the CREATE SCHEMA statement # and it doesn't support multiple schema name. For example, if you set PG_SCHEMA # to something like "user_schema, public", the search path will be set like this # SET search_path = user_schema, public; # forcing the use of an other schema (here user_schema) than the one from Oracle # schema set in the SCHEMA directive. You can also set the default search_path # for the PostgreSQL user you are using to connect to the destination database # by using: # ALTER ROLE username SET search_path TO user_schema, public; #in this case you don't have to set PG_SCHEMA. #PG_SCHEMA # Use this directive to add a specific schema to the search path to look # for PostGis functions. #POSTGIS_SCHEMA # Allow to add a comma separated list of system user to exclude from # Oracle extraction. Oracle have many of them following the modules # installed. By default it will suppress all object owned by the following # system users: # 'SYSTEM','CTXSYS','DBSNMP','EXFSYS','LBACSYS','MDSYS','MGMT_VIEW', # 'OLAPSYS','ORDDATA','OWBSYS','ORDPLUGINS','ORDSYS','OUTLN', # 'SI_INFORMTN_SCHEMA','SYS','SYSMAN','WK_TEST','WKSYS','WKPROXY', # 'WMSYS','XDB','APEX_PUBLIC_USER','DIP','FLOWS_020100','FLOWS_030000', # 'FLOWS_040100','FLOWS_010600','FLOWS_FILES','MDDATA','ORACLE_OCM', # 'SPATIAL_CSW_ADMIN_USR','SPATIAL_WFS_ADMIN_USR','XS\$NULL','PERFSTAT', # 'SQLTXPLAIN','DMSYS','TSMSYS','WKSYS','APEX_040000','APEX_040200', # 'DVSYS','OJVMSYS','GSMADMIN_INTERNAL','APPQOSSYS','DVSYS','DVF', # 'AUDSYS','APEX_030200','MGMT_VIEW','ODM','ODM_MTR','TRACESRV','MTMSYS', # 'OWBSYS_AUDIT','WEBSYS','WK_PROXY','OSE\$HTTP\$ADMIN', # 'AURORA\$JIS\$UTILITY\$','AURORA\$ORB\$UNAUTHENTICATED', # 'DBMS_PRIVILEGE_CAPTURE','CSMIG','MGDSYS','SDE','DBSFWUSER' # Other list of users set to this directive will be added to this list. #SYSUSERS OE,HR # List of schema to get functions/procedures meta information that are used # in the current schema export. When replacing call to function with OUT # parameters, if a function is declared in an other package then the function # call rewriting can not be done because Ora2Pg only know about functions # declared in the current schema. By setting a comma separated list of schema # as value of this directive, Ora2Pg will look forward in these packages for # all functions/procedures/packages declaration before proceeding to current # schema export. #LOOK_FORWARD_FUNCTION SCOTT,OE # Force Ora2Pg to not look for function declaration. Note that this will prevent # Ora2Pg to rewrite function replacement call if needed. Do not enable it unless # looking forward at function breaks other export. NO_FUNCTION_METADATA 0 #------------------------------------------------------------------------------ # ENCODING SECTION (Define client encoding at Oracle and PostgreSQL side) #------------------------------------------------------------------------------ # Enforce default language setting following the Oracle database encoding. This # may be used with multibyte characters like UTF8. Here are the default values # used by Ora2Pg, you may not change them unless you have problem with this # encoding. This will set \$ENV{NLS_LANG} to the given value. #NLS_LANG AMERICAN_AMERICA.AL32UTF8 # This will set \$ENV{NLS_NCHAR} to the given value. #NLS_NCHAR AL32UTF8 # By default PostgreSQL client encoding is automatically set to UTF8 to avoid # encoding issue. If you have changed the value of NLS_LANG you might have to # change the encoding of the PostgreSQL client. #CLIENT_ENCODING UTF8 # To force utf8 encoding of the PL/SQL code exported, enable this directive. # Could be helpful in some rare condition. FORCE_PLSQL_ENCODING 0 #------------------------------------------------------------------------------ # EXPORT SECTION (Export type and filters) #------------------------------------------------------------------------------ # Type of export. Values can be the following keyword: # TABLE Export tables, constraints, indexes, ... # PACKAGE Export packages # INSERT Export data from table as INSERT statement # COPY Export data from table as COPY statement # VIEW Export views # GRANT Export grants # SEQUENCE Export sequences # TRIGGER Export triggers # FUNCTION Export functions # PROCEDURE Export procedures # TABLESPACE Export tablespace (PostgreSQL >= 8 only) # TYPE Export user defined Oracle types # PARTITION Export range or list partition (PostgreSQL >= v8.4) # FDW Export table as foreign data wrapper tables # MVIEW Export materialized view as snapshot refresh view # QUERY Convert Oracle SQL queries from a file. # KETTLE Generate XML ktr template files to be used by Kettle. # DBLINK Generate oracle foreign data wrapper server to use as dblink. # SYNONYM Export Oracle's synonyms as views on other schema's objects. # DIRECTORY Export Oracle's directories as external_file extension objects. # LOAD Dispatch a list of queries over multiple PostgreSQl connections. # TEST perform a diff between Oracle and PostgreSQL database. # TEST_COUNT perform only a row count between Oracle and PostgreSQL tables. # TEST_VIEW perform a count on both side of number of rows returned by views # TEST_DATA perform data validation check on rows at both sides. # SEQUENCE_VALUES export DDL to set the last values of sequences TYPE TABLE # Set this to 1 if you don't want to export comments associated to tables and # column definitions. Default is enabled. DISABLE_COMMENT 0 # Set which object to export from. By default Ora2Pg export all objects. # Value must be a list of object name or regex separated by space. Note # that regex will not works with 8i database, use % placeholder instead # Ora2Pg will use the LIKE operator. There is also some extended use of # this directive, see chapter "Limiting object to export" in documentation. #ALLOW TABLE_TEST # Set which object to exclude from export process. By default none. Value # must be a list of object name or regexp separated by space. Note that regex # will not works with 8i database, use % placeholder instead Ora2Pg will use # the NOT LIKE operator. There is also some extended use of this directive, # see chapter "Limiting object to export" in documentation. #EXCLUDE OTHER_TABLES # By default Ora2Pg exclude from export some Oracle "garbage" tables that should # never be part of an export. This behavior generates a lot of REGEXP_LIKE # expressions which are slowing down the export when looking at tables. To disable # this behavior enable this directive, you will have to exclude or clean up later # by yourself the unwanted tables. The regexp used to exclude the table are # defined in the array @EXCLUDED_TABLES in lib/Ora2Pg.pm. Note this is behavior # is independant to the EXCLUDE configuration directive. NO_EXCLUDED_TABLE 0 # Set which view to export as table. By default none. Value must be a list of # view name or regexp separated by space. If the object name is a view and the # export type is TABLE, the view will be exported as a create table statement. # If export type is COPY or INSERT, the corresponding data will be exported. #VIEW_AS_TABLE VIEW_NAME # Set which materialized view to export as table. By default none. Value must # be a list of materialized view name or regexp separated by space. If the # object name is a materialized view and the export type is TABLE, the view # will be exported as a create table statement. If export type is COPY or # INSERT, the corresponding data will be exported. #MVIEW_AS_TABLE VIEW_NAME # By default Ora2Pg try to order views to avoid error at import time with # nested views. With a huge number of view this can take a very long time, # you can bypass this ordering by enabling this directive. NO_VIEW_ORDERING 0 # When exporting GRANTs you can specify a comma separated list of objects # for which privilege will be exported. Default is export for all objects. # Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, # PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY. Only one object # type is allowed at a time. For example set it to TABLE if you just want to # export privilege on tables. You can use the -g option to overwrite it. # When used this directive prevent the export of users unless it is set to # USER. In this case only users definitions are exported. #GRANT_OBJECT TABLE # By default Ora2Pg will export your external table as file_fdw tables. If # you don't want to export those tables at all, set the directive to 0. EXTERNAL_TO_FDW 1 # Add a TRUNCATE TABLE instruction before loading data on COPY and INSERT # export. When activated, the instruction will be added only if there's no # global DELETE clause or one specific to the current table (see bellow). TRUNCATE_TABLE 0 # Support for include a DELETE FROM ... WHERE clause filter before importing # data and perform a delete of some lines instead of truncatinf tables. # Value is construct as follow: TABLE_NAME[DELETE_WHERE_CLAUSE], or # if you have only one where clause for all tables just put the delete # clause as single value. Both are possible too. Here are some examples: #DELETE 1=1 # Apply to all tables and delete all tuples #DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST #DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] # The last applies two different delete where clause on tables TABLE_TEST and # TABLE_INFO and a generic delete where clause on DATE_CREATE to all other tables. # If TRUNCATE_TABLE is enabled it will be applied to all tables not covered by # the DELETE definition. # When enabled this directive forces ora2pg to export all tables, index # constraints, and indexes using the tablespace name defined in Oracle database. # This works only with tablespaces that are not TEMP, USERS and SYSTEM. USE_TABLESPACE 0 # Enable this directive to reorder columns and minimized the footprint # on disk, so that more rows fit on a data page, which is the most important # factor for speed. Default is same order than in Oracle table definition, # that should be enough for most usage. REORDERING_COLUMNS 0 # Support for include a WHERE clause filter when dumping the contents # of tables. Value is construct as follow: TABLE_NAME[WHERE_CLAUSE], or # if you have only one where clause for each table just put the where # clause as value. Both are possible too. Here are some examples: #WHERE 1=1 # Apply to all tables #WHERE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST #WHERE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] # The last applies two different where clause on tables TABLE_TEST and # TABLE_INFO and a generic where clause on DATE_CREATE to all other tables # Sometime you may want to extract data from an Oracle table but you need a # a custom query for that. Not just a "SELECT * FROM table" like Ora2Pg does # but a more complex query. This directive allows you to override the query # used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. # If you have multiple tables to extract by replacing the Ora2Pg query, you can # define multiple REPLACE_QUERY lines. #REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] # To add a DROP IF EXISTS before creating the object, enable # this directive. Can be useful in an iterative work. Default is disabled. DROP_IF_EXISTS 0 # PostgreSQL do not supports Global Temporary Table natively but you can use # the pgtt extension to emulate this behavior. Enable this directive to export # global temporary table. EXPORT_GTT 0 # By default the pgtt extension is load using the superuser privilege, set it # to 1 if you run the SQL scripts generated using a non superuser user. It will # use LOAD '\$libdir/plugins/pgtt' instead of LOAD 'pgtt'. PGTT_NOSUPERUSER 0 #------------------------------------------------------------------------------ # FULL TEXT SEARCH SECTION (Control full text search export behaviors) #------------------------------------------------------------------------------ # Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using # pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes # and CTXCAT indexes using pg_trgm. Most of the time using pg_trgm is enough, # this is why this directive stand for. # CONTEXT_AS_TRGM 0 # By default Ora2Pg creates a function-based index to translate Oracle Text # indexes. # CREATE INDEX ON t_document # USING gin(to_tsvector('french', title)); # You will have to rewrite the CONTAIN() clause using to_tsvector(), example: # SELECT id,title FROM t_document # WHERE to_tsvector(title)) @@ to_tsquery('search_word'); # # To force Ora2Pg to create an extra tsvector column with a dedicated triggers # for FTS indexes, disable this directive. In this case, Ora2Pg will add the # column as follow: ALTER TABLE t_document ADD COLUMN tsv_title tsvector; # Then update the column to compute FTS vectors if data have been loaded before # UPDATE t_document SET tsv_title = # to_tsvector('french', coalesce(title,'')); # To automatically update the column when a modification in the title column # appears, Ora2Pg adds the following trigger: # # CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS \$\$ # BEGIN # IF TG_OP = 'INSERT' OR new.title != old.title THEN # new.tsv_title := # to_tsvector('french', coalesce(new.title,'')); # END IF; # return new; # END # \$\$ LANGUAGE plpgsql; # CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE # ON t_document # FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); # # When the Oracle text index is defined over multiple column, Ora2Pg will use # setweight() to set a weight in the order of the column declaration. # FTS_INDEX_ONLY 1 # Use this directive to force text search configuration to use. When it is not # set, Ora2Pg will autodetect the stemmer used by Oracle for each index and # pg_catalog.english if nothing is found. # #FTS_CONFIG pg_catalog.french # If you want to perform your text search in an accent insensitive way, enable # this directive. Ora2Pg will create an helper function over unaccent() and # creates the pg_trgm indexes using this function. With FTS Ora2Pg will # redefine your text search configuration, for example: # # CREATE TEXT SEARCH CONFIGURATION fr (COPY = pg_catalog.french); # ALTER TEXT SEARCH CONFIGURATION fr # ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; # # When enabled, Ora2pg will create the wrapper function: # # CREATE OR REPLACE FUNCTION unaccent_immutable(text) # RETURNS text AS # \$\$ # SELECT public.unaccent('public.unaccent', $1) # \$\$ LANGUAGE sql IMMUTABLE # COST 1; # # indexes are exported as follow: # # CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document # USING gin (unaccent_immutable(title) gin_trgm_ops); # # In your queries you will need to use the same function in the search to # be able to use the function-based index. Example: # # SELECT * FROM t_document # WHERE unaccent_immutable(title) LIKE '%donnees%'; # USE_UNACCENT 0 # Same as above but call lower() in the unaccent_immutable() function: # # CREATE OR REPLACE FUNCTION unaccent_immutable(text) # RETURNS text AS # \$\$ # SELECT lower(public.unaccent('public.unaccent', $1)); # \$\$ LANGUAGE sql IMMUTABLE; # USE_LOWER_UNACCENT 0 #------------------------------------------------------------------------------ # DATA DIFF SECTION (only delete and insert actually changed rows) #------------------------------------------------------------------------------ # EXPERIMENTAL! Not yet working correctly with partitioned tables, parallelism, # and direct Postgres connection! Test before using in production! # This feature affects SQL output for data (INSERT or COPY). # The deletion and (re-)importing of data is redirected to temporary tables # (with configurable suffix) and matching entries (i.e. quasi-unchanged rows) # eliminated before actual application of the DELETE, UPDATE and INSERT. # Optional functions can be specified that are called before or after the # actual DELETE, UPDATE and INSERT per table, or after all tables have been # processed. # # Enable DATADIFF functionality DATADIFF 0 # Use UPDATE where changed columns can be matched by the primary key # (otherwise rows are DELETEd and re-INSERTed, which may interfere with # inverse foreign keys relationships!) DATADIFF_UPDATE_BY_PKEY 0 # Suffix for temporary tables holding rows to be deleted and to be inserted. # Pay attention to your tables names: # 1) There better be no two tables with names such that name1 + suffix = name2 # 2) length(suffix) + length(tablename) < NAMEDATALEN (usually 64) DATADIFF_DEL_SUFFIX _del DATADIFF_UPD_SUFFIX _upd DATADIFF_INS_SUFFIX _ins # Allow setting the work_mem and temp_buffers parameters # to keep temp tables in memory and have efficient sorting, etc. DATADIFF_WORK_MEM 256 MB DATADIFF_TEMP_BUFFERS 512 MB # The following are names of functions that will be called (via SELECT) # after the temporary tables have been reduced (by removing matching rows) # and right before or right after the actual DELETE and INSERT are performed. # They must take four arguments, which should ideally be of type "regclass", # representing the real table, the "deletions", the "updates", and the # "insertions" temp table names, respectively. They are called before # re-activation of triggers, indexes, etc. (if configured). #DATADIFF_BEFORE my_datadiff_handler_function #DATADIFF_AFTER my_datadiff_handler_function # Another function can be called (via SELECT) right before the entire COMMIT # (i.e., after re-activation of indexes, triggers, etc.), which will be # passed in Postgres ARRAYs of the table names of the real tables, the # "deletions", the "updates" and the "insertions" temp tables, respectively, # with same array index positions belonging together. So this function should # take four arguments of type regclass[] #DATADIFF_AFTER_ALL my_datadiff_bunch_handler_function # If in doubt, use schema-qualified function names here. # The search_path will have been set to PG_SCHEMA if EXPORT_SCHEMA == 1 # (as defined by you in those config parameters, see above), # i.e., the "public" schema is not contained if EXPORT_SCHEMA == 1 #------------------------------------------------------------------------------ # CONSTRAINT SECTION (Control constraints export and import behaviors) #------------------------------------------------------------------------------ # Support for turning off certain schema features in the postgres side # during schema export. Values can be : fkeys, pkeys, ukeys, indexes, checks # separated by a space character. # fkeys : turn off foreign key constraints # pkeys : turn off primary keys # ukeys : turn off unique column constraints # indexes : turn off all other index types # checks : turn off check constraints #SKIP fkeys pkeys ukeys indexes checks # By default names of the primary and unique key in the source Oracle database # are ignored and key names are autogenerated in the target PostgreSQL database # with the PostgreSQL internal default naming rules. If you want to preserve # Oracle primary and unique key names set this option to 1. # Please note if value of USE_TABLESPACE is set to 1 the value of this option is # enforced to 1 to preserve correct primary and unique key allocation to tablespace. KEEP_PKEY_NAMES 0 # Enable this directive if you want to add primary key definitions inside the # create table statements. If disabled (the default) primary key definition # will be added with an alter table statement. Enable it if you are exporting # to GreenPlum PostgreSQL database. PKEY_IN_CREATE 0 # This directive allow you to add an ON UPDATE CASCADE option to a foreign # key when a ON DELETE CASCADE is defined or always. Oracle do not support # this feature, you have to use trigger to operate the ON UPDATE CASCADE. # As PostgreSQL has this feature, you can choose how to add the foreign # key option. There is three value to this directive: never, the default # that mean that foreign keys will be declared exactly like in Oracle. # The second value is delete, that mean that the ON UPDATE CASCADE option # will be added only if the ON DELETE CASCADE is already defined on the # foreign Keys. The last value, always, will force all foreign keys to be # defined using the update option. FKEY_ADD_UPDATE never # When exporting tables, Ora2Pg normally exports constraints as they are; # if they are non-deferrable they are exported as non-deferrable. # However, non-deferrable constraints will probably cause problems when # attempting to import data to PostgreSQL. The following option set to 1 # will cause all foreign key constraints to be exported as deferrable FKEY_DEFERRABLE 0 # In addition when exporting data the DEFER_FKEY option set to 1 will add # a command to defer all foreign key constraints during data export and # the import will be done in a single transaction. This will work only if # foreign keys have been exported as deferrable and you are not using direct # import to PostgreSQL (PG_DSN is not defined). Constraints will then be # checked at the end of the transaction. This directive can also be enabled # if you want to force all foreign keys to be created as deferrable and # initially deferred during schema export (TABLE export type). DEFER_FKEY 0 # If deferring foreign keys is not possible due to the amount of data in a # single transaction, you've not exported foreign keys as deferrable or you # are using direct import to PostgreSQL, you can use the DROP_FKEY directive. # It will drop all foreign keys before all data import and recreate them at # the end of the import. DROP_FKEY 0 #------------------------------------------------------------------------------ # TRIGGERS AND SEQUENCES SECTION (Control triggers and sequences behaviors) #------------------------------------------------------------------------------ # Disables alter of sequences on all tables in COPY or INSERT mode. # Set to 1 if you want to disable update of sequence during data migration. DISABLE_SEQUENCE 0 # Disables triggers on all tables in COPY or INSERT mode. Available modes # are USER (user defined triggers) and ALL (includes RI system # triggers). Default is 0 do not add SQL statement to disable trigger. # If you want to disable triggers during data migration, set the value to # USER if your are connected as non superuser and ALL if you are connected # as PostgreSQL superuser. A value of 1 is equal to USER. DISABLE_TRIGGERS 0 #------------------------------------------------------------------------------ # OBJECT MODIFICATION SECTION (Control objects structure or name modifications) #------------------------------------------------------------------------------ # You may wish to just extract data from some fields, the following directives # will help you to do that. Works only with export type INSERT or COPY # Modify output from the following tables(fields separate by space or comma) #MODIFY_STRUCT TABLE_TEST(dico,dossier) # When there is a lot of columns dropped in the target database compared to the # Oracle database, being able to exclude columns from data export will simplify # the configuration compared to MODIFY_STRUCT. The following directive can be # used to define the columns per table that must be excluded from data export. #EXCLUDE_COLUMNS TABLE_TEST(dropcol1,dropcol2) # You may wish to change table names during data extraction, especally for # replication use. Give a list of tables separate by space as follow. #REPLACE_TABLES ORIG_TB_NAME1:NEW_TB_NAME1 ORIG_TB_NAME2:NEW_TB_NAME2 # You may wish to change column names during export. Give a list of tables # and columns separate by comma as follow. #REPLACE_COLS TB_NAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) # By default all object names are converted to lower case, if you # want to preserve Oracle object name as-is set this to 1. Not recommended # unless you always quote all tables and columns on all your scripts. PRESERVE_CASE 0 # Add the given value as suffix to index names. Useful if you have indexes # with same name as tables. Not so common but it can help. #INDEXES_SUFFIX _idx # Enable this directive to rename all indexes using tablename_columns_names. # Could be very useful for database that have multiple time the same index name # or that use the same name than a table, which is not allowed by PostgreSQL # Disabled by default. INDEXES_RENAMING 0 # Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops # support B-tree indexes on the corresponding types. The difference from the # default operator classes is that the values are compared strictly character by # character rather than according to the locale-specific collation rules. This # makes these operator classes suitable for use by queries involving pattern # matching expressions (LIKE or POSIX regular expressions) when the database # does not use the standard "C" locale. If you enable, with value 1, this will # force Ora2Pg to export all indexes defined on varchar2() and char() columns # using those operators. If you set it to a value greater than 1 it will only # change indexes on columns where the charactere limit is greater or equal than # this value. For example, set it to 128 to create these kind of indexes on # columns of type varchar2(N) where N >= 128. USE_INDEX_OPCLASS 0 # Enable this directive if you want that your partition tables will be # renamed. Disabled by default. If you have multiple partitioned table, # when exported to PostgreSQL some partitions could have the same name # but different parent tables. This is not allowed, table name must be # unique, in this case enable this directive. RENAME_PARTITION 0 # If you don't want to reproduce the partitioning like in Oracle and want to # export all partitionned Oracle data into the main single table in PostgreSQL # enable this directive. Ora2Pg will export all data into the main table name. # Default is to use partitionning, Ora2Pg will export data from each partition # and import them into the PostgreSQL dedicated partition table. DISABLE_PARTITION 0 # How to export partition by reference. Possible values are none, duplicate or # the number of hash partition to create. Default is 'none' to not export the # partitions by reference. PARTITION_BY_REFERENCE none # Activating this directive will force Ora2Pg to add WITH (OIDS) when creating # tables or views as tables. Default is same as PostgreSQL, disabled. WITH_OID 0 # Allow escaping of column name using Oracle reserved words. ORA_RESERVED_WORDS audit,comment,references # Enable this directive if you have tables or column names that are a reserved # word for PostgreSQL. Ora2Pg will double quote the name of the object. USE_RESERVED_WORDS 0 # By default Ora2Pg export Oracle tables with the NOLOGGING attribute as # UNLOGGED tables. You may want to fully disable this feature because # you will lost all data from unlogged table in case of PostgreSQL crash. # Set it to 1 to export all tables as normal table. DISABLE_UNLOGGED 0 # Increase varchar max character constraints to support PostgreSQL two bytes # character encoding when the source database applies the length constraint # on characters not bytes. Default disabled. DOUBLE_MAX_VARCHAR 0 #------------------------------------------------------------------------------ # OUTPUT SECTION (Control output to file or PostgreSQL database) #------------------------------------------------------------------------------ # Define the following directive to send export directly to a PostgreSQL # database, this will disable file output. Note that these directives are only # used for data export, other export need to be imported manually through the # use of psql or any other PostgreSQL client. #PG_DSN dbi:Pg:dbname=test_db;host=localhost;port=5432 #PG_USER test #PG_PWD test # By default all output is dump to STDOUT if not send directly to postgresql # database (see above). Give a filename to save export to it. If you want # a Gzip'd compressed file just add the extension .gz to the filename (you # need perl module Compress::Zlib from CPAN). Add extension .bz2 to use Bzip2 # compression. OUTPUT output.sql # Base directory where all dumped files must be written #OUTPUT_DIR /var/tmp # Path to the bzip2 program. See OUTPUT directive above. BZIP2 $bzip2 # Allow object constraints to be saved in a separate file during schema export. # The file will be named CONSTRAINTS_OUTPUT. Where OUTPUT is the value of the # corresponding configuration directive. You can use .gz xor .bz2 extension to # enable compression. Default is to save all data in the OUTPUT file. This # directive is usable only with TABLE export type. FILE_PER_CONSTRAINT 0 # Allow indexes to be saved in a separate file during schema export. The file # will be named INDEXES_OUTPUT. Where OUTPUT is the value of the corresponding # configuration directive. You can use the .gz, .xor, or .bz2 file extension to # enable compression. Default is to save all data in the OUTPUT file. This # directive is usable only with TABLE or TABLESPACE export type. With the # TABLESPACE export, it is used to write "ALTER INDEX ... TABLESPACE ..." into # a separate file named TBSP_INDEXES_OUTPUT that can be loaded at end of the # migration after the indexes creation to move the indexes. FILE_PER_INDEX 0 # Allow foreign key declaration to be saved in a separate file during # schema export. By default foreign keys are exported into the main # output file or in the CONSTRAINT_output.sql file. When enabled foreign # keys will be exported into a file named FKEYS_output.sql FILE_PER_FKEYS 0 # Allow data export to be saved in one file per table/view. The files # will be named as tablename_OUTPUT. Where OUTPUT is the value of the # corresponding configuration directive. You can use .gz xor .bz2 # extension to enable compression. Default is to save all data in one # file. This is usable only during INSERT or COPY export type. FILE_PER_TABLE 0 # Allow function export to be saved in one file per function/procedure. # The files will be named as funcname_OUTPUT. Where OUTPUT is the value # of the corresponding configuration directive. You can use .gz xor .bz2 # extension to enable compression. Default is to save all data in one # file. It is usable during FUNCTION, PROCEDURE, TRIGGER and PACKAGE # export type. FILE_PER_FUNCTION 0 # By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through # a call to the Perl pragma: # # use open ':utf8'; # # You can override this encoding by using the BINMODE directive, for example you # can set it to :locale to use your locale or iso-8859-7, it will respectively use # # use open ':locale'; # use open ':encoding(iso-8859-7)'; # # If you have change the NLS_LANG in non UTF8 encoding, you might want to set this # directive. See http://perldoc.perl.org/5.14.2/open.html for more information. # Most of the time, you might leave this directive commented. #BINMODE utf8 # Set it to 0 to not include the call to \\set ON_ERROR_STOP ON in all SQL # scripts. By default this order is always present. STOP_ON_ERROR 1 # Enable this directive to use COPY FREEZE instead of a simple COPY to # export data with rows already frozen. This is intended as a performance # option for initial data loading. Rows will be frozen only if the table # being loaded has been created or truncated in the current subtransaction. # This will only works with export to file and when -J or ORACLE_COPIES is # not set or default to 1. It can be used with direct import into PostgreSQL # under the same condition but -j or JOBS must also be unset or default to 1. COPY_FREEZE 0 # By default Ora2Pg use CREATE OR REPLACE in functions and views DDL, if you # need not to override existing functions or views disable this configuration # directive, DDL will not include OR REPLACE. CREATE_OR_REPLACE 1 # This directive can be used to send an initial command to PostgreSQL, just # after the connection. For example to set some session parameters. This # directive can be used multiple time. #PG_INITIAL_COMMAND # Add an ON CONFLICT DO NOTHING to all INSERT statements generated for this # type of data export. INSERT_ON_CONFLICT 0 #------------------------------------------------------------------------------ # TYPE SECTION (Control type behaviors and redefinitions) #------------------------------------------------------------------------------ # If you're experiencing problems in data type export, the following directive # will help you to redefine data type translation used in Ora2pg. The syntax is # a comma separated list of "Oracle datatype:Postgresql data type". Here are the # data type that can be redefined and their default value. If you want to # replace a type with a precision and scale you need to escape the coma with # a backslash. For example, if you want to replace all NUMBER(*,0) into bigint # instead of numeric(38)add the following: # DATA_TYPE NUMBER(*\\,0):bigint # Here is the default replacement for all Oracle's types. You don't have to # recopy all type conversion but just the one you want to rewrite. #DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,NVARCHAR:varchar,NCHAR:char,DATE:timestamp(0),LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW(16):uuid,RAW(32):uuid,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:integer,INTEGER:integer,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone # If set to 1 replace portable numeric type into PostgreSQL internal type. # Oracle data type NUMBER(p,s) is approximatively converted to real and # float PostgreSQL data type. If you have monetary fields or don't want # rounding issues with the extra decimals you should preserve the same # numeric(p,s) PostgreSQL data type. Do that only if you need exactness # because using numeric(p,s) is slower than using real or double. PG_NUMERIC_TYPE 1 # If set to 1 replace portable numeric type into PostgreSQL internal type. # Oracle data type NUMBER(p) or NUMBER are converted to smallint, integer # or bigint PostgreSQL data type following the length of the precision. If # NUMBER without precision are set to DEFAULT_NUMERIC (see bellow). PG_INTEGER_TYPE 1 # NUMBER() without precision are converted by default to bigint only if # PG_INTEGER_TYPE is true. You can overwrite this value to any PG type, # like integer or float. DEFAULT_NUMERIC bigint # Set it to 0 if you don't want to export milliseconds from Oracle timestamp # columns. Timestamp will be formated with to_char(..., 'YYYY-MM-DD HH24:MI:SS') # Enabling this directive, the default, format is 'YYYY-MM-DD HH24:MI:SS.FF'. ENABLE_MICROSECOND 1 # If you want to replace some columns as PostgreSQL boolean define here a list # of tables and column separated by space as follows. You can also give a type # and a precision to automatically convert all fields of that type as a boolean. # For example: NUMBER:1 or CHAR:1 will replace any field of type number(1) or # char(1) as a boolean in all exported tables. #REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 # Use this to add additional definitions of the possible boolean values in Oracle # field. You must set a space separated list of TRUE:FALSE values. BY default: #BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled # When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL. # This could be a problem if your column is defined with NOT NULL constraint. # If you can not remove the constraint, use this directive to set an arbitral # date that will be used instead. You can also use -INFINITY if you don't want # to use a fake date. #REPLACE_ZERO_DATE 1970-01-01 00:00:00 # Some time you need to force the destination type, for example a column # exported as timestamp by Ora2Pg can be forced into type date. Value is # a comma-separated list of TABLE:COLUMN:TYPE structure. If you need to use # comma or space inside type definition you will have to backslash them. # # MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) # # Type of table1.col3 will be replaced by a varchar and table1.col4 by # a decimal with precision and scale. # # If the column's type is a user defined type Ora2Pg will autodetect the # composite type and will export its data using ROW(). Some Oracle user # defined types are just array of a native type, in this case you may want # to transform this column in simple array of a PostgreSQL native type. # To do so, just redefine the destination type as wanted and Ora2Pg will # also transform the data as an array. For example, with the following # definition in Oracle: # # CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); # CREATE TABLE club (Name VARCHAR2(10), # Address VARCHAR2(20), # City VARCHAR2(20), # Phone VARCHAR2(8), # Members mem_type # ); # # custom type "mem_type" is just a string array and can be translated into # the following in PostgreSQL: # # CREATE TABLE club ( # name varchar(10), # address varchar(20), # city varchar(20), # phone varchar(8), # members text[] # ) ; # # To do so, just use the directive as follow: # # MODIFY_TYPE CLUB:MEMBERS:text[] # # Ora2Pg will take care to transform all data of this column in the correct # format. Only arrays of characters and numerics types are supported. #MODIFY_TYPE # By default Oracle call to function TO_NUMBER will be translated as a cast # into numeric. For example, TO_NUMBER('10.1234') is converted into PostgreSQL # call to_number('10.1234')::numeric. If you want you can cast the call to integer # or bigint by changing the value of the configuration directive. If you need # better control of the format, just set it as value, for example: # TO_NUMBER_CONVERSION 99999999999999999999.9999999999 # will convert the code above as: # TO_NUMBER('10.1234', '99999999999999999999.9999999999') # Any value of the directive that it is not numeric, integer or bigint will # be taken as a mask format. If set to none, no conversion will be done. TO_NUMBER_CONVERSION numeric # By default varchar2 without size constraint are tranlated into text. If you # want to keep the varchar name, disable this directive. VARCHAR_TO_TEXT 1 # Usually identity column must be bigint to correspond to an auto increment # sequence. If, for any reason you want Ora2Pg respect the DATA_TYPE you have # set, disable this directive. FORCE_IDENTITY_BIGINT 1 # Remove timezone part into the format of the TO_CHAR() function TO_CHAR_NOTIMEZONE 1 #------------------------------------------------------------------------------ # GRANT SECTION (Control priviledge and owner export) #------------------------------------------------------------------------------ # Set this to 1 to replace default password for all extracted user # during GRANT export GEN_USER_PWD 0 # By default the owner of database objects is the one you're using to connect # to PostgreSQL. If you use an other user (e.g. postgres) you can force # Ora2Pg to set the object owner to be the one used in the Oracle database by # setting the directive to 1, or to a completely different username by setting # the directive value # to that username. FORCE_OWNER 0 # Ora2Pg use the function's security privileges set in Oracle and it is often # defined as SECURITY DEFINER. If you want to override those security privileges # for all functions and use SECURITY DEFINER instead, enable this directive. FORCE_SECURITY_INVOKER 0 #------------------------------------------------------------------------------ # DATA SECTION (Control data export behaviors) #------------------------------------------------------------------------------ # Extract data by bulk of DATA_LIMIT tuples at once. Default 10000. If you set # a high value be sure to have enough memory if you have million of rows. DATA_LIMIT $DATA_LIMIT_DEFAULT # When Ora2Pg detect a table with some BLOB it will automatically reduce the # value of this directive by dividing it by 10 until his value is below 1000. # You can control this value by setting BLOB_LIMIT. Exporting BLOB use lot of # ressources, setting it to a too high value can produce OOM. #BLOB_LIMIT 500 # Apply same behavior on CLOB than BLOB with BLOB_LIMIT settings. This is # especially useful if you have large CLOB data. CLOB_AS_BLOB 1 # By default all data that are not of type date or time are escaped. If you # experience any problem with that you can set it to 1 to disable it. This # directive is only used during a COPY export type. # See STANDARD_CONFORMING_STRINGS for enabling/disabling escape with INSERT # statements. NOESCAPE 0 # This directive may be used if you want to change the default isolation # level of the data export transaction. Default is now to set the level # to a serializable transaction to ensure data consistency. Here are the # allowed value of this directive: readonly, readwrite, serializable and # committed (read committed). TRANSACTION serializable # This controls whether ordinary string literals ('...') treat backslashes # literally, as specified in SQL standard. This was the default before Ora2Pg # v8.5 so that all strings was escaped first, now this is currently on, causing # Ora2Pg to use the escape string syntax (E'...') if this parameter is not # set to 0. This is the exact behavior of the same option in PostgreSQL. # This directive is only used during INSERT export to build INSERT statements. # See NOESCAPE for enabling/disabling escape in COPY statements. STANDARD_CONFORMING_STRINGS 1 # Use this directive to set the database handle's 'LongReadLen' attribute to # a value that will be the larger than the expected size of the LOB. The default # is 1MB witch may not be enough to extract BLOB objects. If the size of the LOB # exceeds the 'LongReadLen' DBD::Oracle will return a 'ORA-24345: A Truncation' # error. Default: 1023*1024 bytes. Take a look at this page to learn more: # http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs # # Important note: If you increase the value of this directive take care that # DATA_LIMIT will probably needs to be reduced. Even if you only have a 1MB blob # trying to read 10000 of them (the default DATA_LIMIT) all at once will require # 10GB of memory. You may extract data from those table separately and set a # DATA_LIMIT to 500 or lower, otherwise you may experience some out of memory. #LONGREADLEN 1047552 # If you want to bypass the 'ORA-24345: A Truncation' error, set this directive # to 1, it will truncate the data extracted to the LongReadLen value. #LONGTRUNCOK 0 # Disable this if you want to load full content of BLOB and CLOB and not use # LOB locators. In this case you will have to set LONGREADLEN to the right # value. Note that this will not improve speed of BLOB export as most of the time is always # consumed by the bytea escaping and in this case export is done line by line # and not by chunk of DATA_LIMIT rows. For more information on how it works, see # http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators # Default is enabled, it use LOB locators. USE_LOB_LOCATOR 1 # Oracle recommends reading from and writing to a LOB in batches using a # multiple of the LOB chunk size. This chunk size defaults to 8k (8192). # Recent tests shown that the best performances can be reach with higher # value like 512K or 4Mb. # # A quick benchmark with 30120 rows with different size of BLOB (200x5Mb, # 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, # LONGREADLEN=170Mb and a total table size of 20GB gives: # # no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) # chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) # chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) # chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) # # In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE set # to 4Mb. Dependind of the size of most BLOB you may want to adjust the value # here. For example if you have a majority of small lobs bellow 8K, using 8192 # is better to not waste space. LOB_CHUNK_SIZE 512000 # Force the use of getStringVal() instead of getClobVal() for XML data export. # Default is 1, enabled for backward compatibility. Set here to 0 to use extract # method a la CLOB and export the XML code as it was stored. Note that XML value # extracted with getStringVal() must not exceed VARCHAR2 size limit otherwize # it will return an error. XML_PRETTY 0 # Enable this directive if you want to continue direct data import on error. # When Ora2Pg receives an error in the COPY or INSERT statement from PostgreSQL # it will log the statement to a file called TABLENAME_error.log in the output # directory and continue to next bulk of data. Like this you can try to fix the # statement and manually reload the error log file. Default is disabled: abort # import on error. LOG_ON_ERROR 0 # If you want to convert CHAR(n) from Oracle into varchar(n) or text under # PostgreSQL, you might want to do some triming on the data. By default # Ora2Pg will auto-detect this conversion and remove any withspace at both # leading and trailing position. If you just want to remove the leadings # character, set the value to LEADING. If you just want to remove the trailing # character, set the value to TRAILING. Default value is BOTH. TRIM_TYPE BOTH # The default triming character is space, use the directive bellow if you need # to change the character that will be removed. For example, set it to - if you # have leading - in the char(n) field. To use space as triming charger, comment # this directive, this is the default value. #TRIM_CHAR - # Internal timestamp retrieves from custom type are extracted in the following # format: 01-JAN-77 12.00.00.000000 AM. It is impossible to know the exact century # that must be used, so by default any year below 49 will be added to 2000 # and others to 1900. You can use this directive to change this default value. # this is only relevant if you have user defined type with a column timestamp. INTERNAL_DATE_MAX 49 # Disable this directive if you want to disable check_function_bodies. # # SET check_function_bodies = false; # # It disables validation of the function body string during CREATE FUNCTION. # Default is to use de postgresql.conf setting that enable it by default. FUNCTION_CHECK 1 # Exporting BLOB takes time, in some circumstances you may want to export # all data except the BLOB columns. In this case disable this directive and # the BLOB columns will not be included into data export. Take care that the # target bytea column do not have a NOT NULL constraint. ENABLE_BLOB_EXPORT 1 # Same but for CLOB data export ENABLE_CLOB_EXPORT 1 # By default data export order will be done by sorting on table name. If you # have huge tables at end of alphabetic order and you are using multiprocess # it can be better to set the sort order on size so that multiple small tables # can be processed before the largest tables finish. In this case set this # directive to size. Possible values are name and size. Note that export type # SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or # INSERT export type. If you want to give you custom export order, just give # a filename as value that contains the ordered list of table to export. Must # be a list of one table per line, in uppercase for Oracle. DATA_EXPORT_ORDER name # By default Ora2Pg use \\i psql command to execute generated SQL files # if you want to use a relative path following the script execution file # enabling this option will use \\ir. See psql help for more information. PSQL_RELATIVE_PATH 0 # Number of rows that must be retrieved on both side for data validation. DATA_VALIDATION_ROWS 10000 # Order of rows between both sides are different once the data have been # modified. In this case data must be ordered using a primary key or a # unique index, that mean that a table without such object can not be # compared. If the validation is done just after the data migration without # any data modification the validation can be done on all tables without any # ordering. DATA_VALIDATION_ORDERING 1 # Stop validating data from a table after a certain amount of row mistmatch. # Default is to stop after 10 rows validation errors. DATA_VALIDATION_ERROR 10 # Use this directive to precise which transformation should be applied to a # column when exporting data. Value must be a semicolon separated list of # TABLE[COLUMN_NAME, ] # For example to replace string 'Oracle' by 'PostgreSQL' in a varchar2 column # use the following. #TRANSFORM_VALUE ERROR_LOG_SAMPLE[DBMS_TYPE,regexp_replace("DBMS_TYPE",'Oracle','PostgreSQL')] #or to replace all Oracle char(0) in a string by a space character: #TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), ' ')] #The expression will be applied in the SQL statemeent used to extract data #from the source database. #------------------------------------------------------------------------------ # PERFORMANCES SECTION (Control export/import performances) #------------------------------------------------------------------------------ # This configuration directive adds multiprocess support to COPY, FUNCTION # and PROCEDURE export type, the value is the number of process to use. # Default is to not use multiprocess. This directive is used to set the number # of cores to used to parallelize data import into PostgreSQL. During FUNCTION # or PROCEDURE export type each function will be translated to plpgsql using a # new process, the performances gain can be very important when you have tons # of function to convert. There's no more limitation in parallel processing # than the number of cores and the PostgreSQL I/O performance capabilities. # Doesn't works under Windows Operating System, it is simply disabled. JOBS 1 # Multiprocess support. This directive should defined the number of parallel # connection to Oracle when extracting data. The limit is the number of cores # on your machine. This is useful if Oracle is the bottleneck. Take care that # this directive can only be used if there is a primary / unique key defined # on a numeric column or that a column is defined in DEFINED_PK. ORACLE_COPIES 1 # Multiprocess support. This directive should defined the number of tables # in parallel data extraction. The limit is the number of cores on your machine. # Ora2Pg will open one database connection for each parallel table extraction. # This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS. # Note that this directive when set upper that 1 will also automatically enable # the FILE_PER_TABLE directive if your are exporting to files. PARALLEL_TABLES 1 # You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in each # query used to export data from Oracle by setting a value upper than 1 to # this directive. A value of 0 or 1 disable the use of parallel hint. # Default is disabled. DEFAULT_PARALLELISM_DEGREE 0 # Parallel mode will not be activated if the table have less rows than # this directive. This prevent fork of Oracle process when it is not # necessary. Default is 100K rows. PARALLEL_MIN_ROWS 100000 # Multiprocess support. This directive is used to split the select queries # between the different connections to Oracle if ORA_COPIES is used. Ora2Pg # will extract data with the following prepare statement: # SELECT * FROM TABLE WHERE MOD(COLUMN, \$ORA_COPIES) = ? # Where \$ORA_COPIES is the total number of cores used to extract data and set # with ORA_COPIES directive, and ? is the current core used at execution time. # This means that Ora2Pg needs to know the numeric column to use in this query. # If this column is a real, float, numeric or decimal, you must add the ROUND() # function with the column to round the value to the nearest integer. #DEFINED_PK TABLE:COLUMN TABLE:ROUND(COLUMN) # Enabling this directive force Ora2Pg to drop all indexes on data import # tables, except automatic index on primary key, and recreate them at end # of data import. This may improve speed a lot during a fresh import. DROP_INDEXES 0 # Specifies whether transaction commit will wait for WAL records to be written # to disk before the command returns a "success" indication to the client. This # is the equivalent to set synchronous_commit directive of postgresql.conf file. # This is only used when you load data directly to PostgreSQL, the default is # off to disable synchronous commit to gain speed at writing data. Some modified # versions of PostgreSQL, like Greenplum, do not have this setting, so in this # case set this directive to 1, ora2pg will not try to change the setting. SYNCHRONOUS_COMMIT 0 #------------------------------------------------------------------------------ # PLSQL SECTION (Control SQL and PL/SQL to PLPGSQL rewriting behaviors) #------------------------------------------------------------------------------ # If the above configuration directive is not enough to validate your PL/SQL code # enable this configuration directive to allow export of all PL/SQL code even if # it is marked as invalid. The 'VALID' or 'INVALID' status applies to functions, # procedures, packages, triggers and user defined types. EXPORT_INVALID 0 # Enable PLSQL to PLPSQL conversion. This is a work in progress, feel # free modify/add you own code and send me patches. The code is under # function plsql_toplpgsql in Ora2PG/PLSQL.pm. Default enabled. PLSQL_PGSQL 1 # Ora2Pg can replace all conditions with a test on NULL by a call to the # coalesce() function to mimic the Oracle behavior where empty field are # considered equal to NULL. Ex: (field1 IS NULL) and (field2 IS NOT NULL) will # be replaced by (coalesce(field1::text, '') = '') and (field2 IS NOT NULL AND # field2::text <> ''). You might want this replacement to be sure that your # application will have the same behavior but if you have control on you app # a better way is to change it to transform empty string into NULL because # PostgreSQL makes the difference. NULL_EQUAL_EMPTY 0 # Force empty_clob() and empty_blob() to be exported as NULL instead as empty # string for the first one and \\\\x for the second. If NULL is allowed in your # column this might improve data export speed if you have lot of empty lob. EMPTY_LOB_NULL 1 # If you don't want to export package as schema but as simple functions you # might also want to replace all call to package_name.function_name. If you # disable the PACKAGE_AS_SCHEMA directive then Ora2Pg will replace all call # to package_name.function_name() by package_name_function_name(). Default # is to use a schema to emulate package. PACKAGE_AS_SCHEMA 1 # Enable this directive if the rewrite of Oracle native syntax (+) of # OUTER JOIN is broken. This will force Ora2Pg to not rewrite such code, # default is to try to rewrite simple form of rigth outer join for the # moment. REWRITE_OUTER_JOIN 1 # By default Oracle functions are marked as STABLE as they can not modify data # unless when used in PL/SQL with variable assignment or as conditional # expression. You can force Ora2Pg to create these function as VOLATILE by # disabling the following configuration directive. FUNCTION_STABLE 1 # By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to force # the user to review the logic of the function. Once it is fixed in Oracle # source code or you want to comment this calls enable the following directive COMMENT_COMMIT_ROLLBACK 0 # It is common to see SAVEPOINT call inside PL/SQL procedure together with # a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is enabled you # may want to also comment SAVEPOINT calls, in this case enable it. COMMENT_SAVEPOINT 0 # Ora2Pg replace all string constant during the pl/sql to plpgsql translation, # string constant are all text include between single quote. If you have some # string placeholder used in dynamic call to queries you can set a list of # regexp to be temporary replaced to not break the parser.The list of regexp # must use the semi colon as separator. For exemple: #STRING_CONSTANT_REGEXP # To support the Alternative Quoting Mechanism (''Q'') for String Literals # set the regexp with the text capture to use to extract the text part. # For example with a variable declared as # c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; # the regexp must be: q'{(.*)}' ora2pg use the \$\$ delimiter. #ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' # If you want to use functions defined in the Orafce library and prevent # Ora2Pg to translate call to these function, enable this directive. # The Orafce library can be found here: https://github.com/orafce/orafce # By default Ora2pg rewrite add_month(), add_year(), date_trunc() and # to_char() functions, but you may prefer to use the orafce version of # these function that do not need any code transformation. USE_ORAFCE 0 # Enable translation of autonomous transactions into a wrapper function # using dblink or pg_background extension. If you don't want to use this # translation and just want the function to be exported as a normal one # without the pragma call, disable this directive. AUTONOMOUS_TRANSACTION 1 #------------------------------------------------------------------------------ # ASSESSMENT SECTION (Control migration assessment behaviors) #------------------------------------------------------------------------------ # Activate the migration cost evaluation. Must only be used with SHOW_REPORT, # FUNCTION, PROCEDURE, PACKAGE and QUERY export type. Default is disabled. # Note that enabling this directive will force PLSQL_PGSQL activation. ESTIMATE_COST 0 # Set the value in minutes of the migration cost evaluation unit. Default # is five minutes per unit. COST_UNIT_VALUE 5 # By default when using SHOW_REPORT the migration report is generated as # simple text, enabling this directive will force ora2pg to create a report # in HTML format. DUMP_AS_HTML 0 # Set the total number of tables to display in the Top N per row and size # list in the SHOW_TABLE and SHOW_REPORT output. Default 10. TOP_MAX 10 # Use this directive to redefined the number of human-days limit where the # migration assessment level must switch from B to C. Default is set to 10 # human-days. HUMAN_DAYS_LIMIT 5 # Set the comma separated list of username that must be used to filter # queries from the DBA_AUDIT_TRAIL table. Default is to not scan this # table and to never look for queries. This parameter is used only with # SHOW_REPORT and QUERY export type with no input file for queries. # Note that queries will be normalized before output unlike when a file # is given at input using the -i option or INPUT directive. #AUDIT_USER USERNAME1,USERNAME2 # By default Ora2Pg will convert call to SYS_GUID() Oracle function # with a call to uuid_generate_v4() from uuid-ossp extension. You can # redefined it to use the gen_random_uuid() function from pgcrypto # extension by changing the function name below. #UUID_FUNCTION uuid_generate_v4 #------------------------------------------------------------------------------ # POSTGRESQL FEATURE SECTION (Control which PostgreSQL features are available) #------------------------------------------------------------------------------ # Set the PostgreSQL major version number of the target database. Ex: 9.6 or 10 # Default is current major version at time of a new release. This replace the # old PG_SUPPORTS_* configuration directives. PG_VERSION 15 # Use btree_gin extenstion to create bitmap like index with pg >= 9.4 # You will need to create the extension by yourself: # create extension btree_gin; # Default is to create GIN index, when disabled, a btree index will be created BITMAP_AS_GIN 1 # Use pg_background extension to create an autonomous transaction instead # of using a dblink wrapper. With pg >= 9.5 only, default is to use dblink. PG_BACKGROUND 0 # By default if you have an autonomous transaction translated using dblink # extension instead of pg_background the connection is defined using the # values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override # the connection string use this directive as follow to set the connection # in the autonomous transaction wrapper function. #DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass # Some versions of PostgreSQL like Redshift doesn't support substr() # and it need to be replaced by a call to substring(). In this case, # disable it. PG_SUPPORTS_SUBSTR 1 #------------------------------------------------------------------------------ # SPATIAL SECTION (Control spatial geometry export) #------------------------------------------------------------------------------ # Enable this directive if you want Ora2Pg to detect the real spatial type and # dimensions used in a spatial column. By default Ora2Pg will look at spatial # indexes to see if the layer_gtype and sdo_indx_dims constraint parameters have # been set, otherwise column will be created with the non-constrained "geometry" # type. Enabling this feature will force Ora2Pg to scan a sample of 50000 lines # to look at the GTYPE used. You can increase or reduce the sample by setting # the value of AUTODETECT_SPATIAL_TYPE to the desired number of line. AUTODETECT_SPATIAL_TYPE 1 # Disable this directive if you don't want to automatically convert SRID to # EPSG using the sdo_cs.map_oracle_srid_to_epsg() function. Default: enabled # If the SDO_SRID returned by Oracle is NULL, it will be replaced by the # default value 8307 converted to its EPSG value: 4326 (see DEFAULT_SRID) # If the value is upper than 1, all SRID will be forced to this value, in # this case DEFAULT_SRID will not be used when Oracle returns a null value # and the value will be forced to CONVERT_SRID. # Note that it is also possible to set the EPSG value on Oracle side when # sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to force the value: # Ex: system> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; CONVERT_SRID 1 # Use this directive to override the default EPSG SRID to used: 4326. # Can be overwritten by CONVERT_SRID, see above. DEFAULT_SRID 4326 # This directive can take three values: WKT (default), WKB and INTERNAL. # When it is set to WKT, Ora2Pg will use SDO_UTIL.TO_WKTGEOMETRY() to # extract the geometry data. When it is set to WKB, Ora2Pg will use the # binary output using SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type # are called at Oracle side, they are slow and you can easily reach Out Of # Memory when you have lot of rows. Also WKB is not able to export 3D geometry # and some geometries like CURVEPOLYGON. In this case you may use the INTERNAL # extraction type. It will use a pure Perl library to convert the SDO_GEOMETRY # data into a WKT representation, the translation is done on Ora2Pg side. # This is a work in progress, please validate your exported data geometries # before use. GEOMETRY_EXTRACT_TYPE INTERNAL # Oracle function to use to extract the srid from ST_Geometry meta information ST_SRID_FUNCTION ST_SRID # Oracle function to use to extract the dimension from ST_Geometry meta information ST_DIMENSION_FUNCTION ST_DIMENSION # Oracle function to used to convert an ST_Geometry value into WKB format ST_ASBINARY_FUNCTION ST_ASBINARY # Oracle function to used to convert an ST_Geometry value into WKT format ST_ASTEXT_FUNCTION ST_ASTEXT # Oracle function to use to extract the geometry type from a ST_Geometry column ST_GEOMETRYTYPE_FUNCTION ST_GEOMETRYTYPE #------------------------------------------------------------------------------ # FDW SECTION (Control Foreign Data Wrapper export) #------------------------------------------------------------------------------ # This directive is used to set the name of the foreign data server that is used # in the "CREATE SERVER name FOREIGN DATA WRAPPER oracle_fdw ..." command. This # import data using oracle_fdw. Default is no foreign server defined. # This only concerns export type FDW, COPY and INSERT. For export type FDW the # default value is orcl #FDW_SERVER orcl # Schema where foreign tables for data migration will be created. If you use # several instances of ora2pg for data migration through the foreign data # wrapper, you might need to change the name of the schema for each instance. FDW_IMPORT_SCHEMA ora2pg_fdw_import # The default behaviour of Ora2Pg is to NOT set the "prefetch" option for # oracle_fdw when used for COPY and INSERT. This directive allows the prefetch # to be set. See oracle_fdw documentation for the current default. #ORACLE_FDW_PREFETCH 1000 # When using Ora2Pg COPY with oracle_fdw it is possible to use two different # modes: 1) "local", which uses psql on the host running Ora2Pg for the "TO" # binary stream; 2) "server", which uses PostgreSQL server-side COPY for the # "TO" binary stream. Both modes use psql for the "FROM STDIN BINARY". However, # "local" runs the psql "FROM STDIN BINARY" on host Ora2Pg is run from, whereas # "server" runs the psql "FROM STDIN BINARY" on the PostgreSQL server. "local" # mode should work on any PostgreSQL-based system, including managed offerings, # which are not expected to support use of "server" mode due to permissions. # The default is "local" as this is compatible with more configurations. ORACLE_FDW_COPY_MODE local # When using Ora2Pg COPY with oracle_fdw it is possible to use either BINARY or # CSV data format. BINARY provides better performance, however, requires exact # data type matching between the FDW and destination table. CSV provides # greater flexibiliity with respect to data type matching: if the FDW and # destination data types are functionally-compatible the columns can be copied. # The default is "binary". ORACLE_FDW_COPY_FORMAT binary # By default Ora2Pg drops the temporary schema ora2pg_fdw_import used to import # the Oracle foreign schema before each new import. If you want to preserve # the existing schema because of modifications or the use of a third party # server, disable this directive. DROP_FOREIGN_SCHEMA 1 #------------------------------------------------------------------------------ # MYSQL SECTION (Control MySQL export behavior) #------------------------------------------------------------------------------ # Enable this if double pipe and double ampersand (|| and &&) should not be # taken as equivalent to OR and AND. It depend of the variable \@sql_mode, # Use it only if Ora2Pg fail on auto detecting this behavior. MYSQL_PIPES_AS_CONCAT 0 # Enable this directive if you want EXTRACT() replacement to use the internal # format returned as an integer, for example DD HH24:MM:SS will be replaced # with format; DDHH24MMSS::bigint, this depend of your apps usage. MYSQL_INTERNAL_EXTRACT_FORMAT 0 #------------------------------------------------------------------------------ # SQL Server SECTION (Control MSSQL export behavior) #------------------------------------------------------------------------------ # PostgreSQL has no equivalent to rowversion datatype and feature, if you want # to remove these useless columns, enable this directive. Columns of datatype # 'rowversion' or 'timestamp' will not be exported. DROP_ROWVERSION 0 # Emulate the same behavior of MSSQL with case insensitive search. If the value # is citext it will use the citext data type instead of char/varchar/text in # tables DDL (Ora2Pg will add a CHECK constraint for columns with a precision). # Instead of citext you can also set a collation name that will be used in the # columns definitions. To disable case insensitive search set it to: none. CASE_INSENSITIVE_SEARCH citext # Append a TOP N clause to the SELECT command used to extract the data from # SQL Server. This is the equivalent to a WHERE ROWNUM < 1000 clause for Oracle. SELECT_TOP 1000 }; close(OUTCFG); if ($^O !~ /MSWin32|dos/i) { # Do not replace configuration directory in scripts/ora2pg if this is a RPM build. if (!$ENV{RPM_BUILD_ROOT}) { `perl -p -i -e 's#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = "$CONFDIR/ora2pg.conf";#' scripts/ora2pg`; } else { # Do not include prefix with rpmbuild `perl -p -i -e 's#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = "$RPM_CONFDIR/ora2pg.conf";#' scripts/ora2pg`; } } else { my $tmp_conf = quotemeta($CONFDIR); `perl -p -e "s#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = '$tmp_conf\\\\ora2pg.conf';#" scripts\\ora2pg > scripts\\ora2pg.tmp`; `copy scripts\\ora2pg.tmp scripts\\ora2pg /Y`; } WriteMakefile( 'NAME' => 'Ora2Pg', 'VERSION_FROM' => 'lib/Ora2Pg.pm', 'LICENSE' => 'gpl_3', 'dist' => { 'COMPRESS'=>'gzip -9f', 'SUFFIX' => 'gz', 'ZIP'=>'/usr/bin/zip','ZIPFLAGS'=>'-rl' }, 'AUTHOR' => 'Gilles Darold (gilles _AT_ darold _DOT_ net)', 'ABSTRACT' => 'Oracle to PostgreSQL migration toolkit', 'EXE_FILES' => [ qw(scripts/ora2pg scripts/ora2pg_scanner) ], 'MAN3PODS' => { 'doc/Ora2Pg.pod' => 'blib/man3/ora2pg.3' }, 'DESTDIR' => $PREFIX, 'INSTALLDIRS' => $ENV{INSTALLDIRS}, 'clean' => {FILES => "$DEST_CONF_FILE lib/blib/"}, 'PREREQ_PM' => {DBI => 0}, 'META_MERGE' => { resources => { homepage => 'http://ora2pg.darold.net/', repository => { type => 'git', git => 'git@github.com:darold/ora2pg.git', web => 'https://github.com/darold/ora2pg', }, }, } ); sub MY::install { my $self = shift; my $string = $self->MM::install; $string =~ s/(pure_install\s+)(.*)/$1 install_all $2/; return $string; } sub MY::postamble { my $postamble = qq{ install_all : \@echo "Installing default configuration file ($DEST_CONF_FILE) to $CONFDIR" \@\$(MKPATH) $CONFDIR \@\$(CP) -f $DEST_CONF_FILE $CONFDIR/$DEST_CONF_FILE \@\$(MKPATH) $DOCDIR \@\$(CP) -f README $DOCDIR/README \@\$(CP) -f INSTALL $DOCDIR/INSTALL \@\$(CP) -f changelog $DOCDIR/changelog }; if ($^O =~ /MSWin32|dos/i) { my $tmp_conf = quotemeta($CONFDIR); $postamble = qq{ install_all : \@echo "Installing default configuration file ($DEST_CONF_FILE) to $CONFDIR" \@\$(MKPATH) $CONFDIR \@\$(CP) $DEST_CONF_FILE $CONFDIR\\$DEST_CONF_FILE \@\$(CP) README $CONFDIR\\README \@\$(CP) INSTALL $CONFDIR\\INSTALL \@\$(CP) changelog $CONFDIR\\changelog }; } return $postamble; } if (!$ENV{QUIET}) { print qq{ Done... ------------------------------------------------------------------------------ Please read documentation at http://ora2pg.darold.net/ before asking for help ------------------------------------------------------------------------------ }; if ($^O !~ /MSWin32|dos/i) { print "Now type: make && make install\n"; } else { print "Now type: dmake && dmake install\n"; } } ora2pg-25.0/README000066400000000000000000005212521500113072400135120ustar00rootroot00000000000000NAME Ora2Pg - Oracle to PostgreSQL database schema converter DESCRIPTION Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scans it automatically and extracts its structure or data, then generates SQL scripts that you can load into your PostgreSQL database. Ora2Pg can be used for anything from reverse engineering Oracle database to huge enterprise database migration or simply replicating some Oracle data into a PostgreSQL database. It is really easy to use and doesn't require any Oracle database knowledge other than providing the parameters needed to connect to the Oracle database. FEATURES Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), the only thing you have to modify is the configuration file ora2pg.conf by setting the DSN to the Oracle database and optionally the name of a schema. Once that's done you just have to set the type of export you want: TABLE with constraints, VIEW, MVIEW, TABLESPACE, SEQUENCE, INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, PACKAGE, PARTITION, TYPE, INSERT or COPY, FDW, QUERY, KETTLE, SYNONYM. By default Ora2Pg exports to a file that you can load into PostgreSQL with the psql client, but you can also import directly into a PostgreSQL database by setting its DSN into the configuration file. With all configuration options of ora2pg.conf you have full control of what should be exported and how. Features included: - Export full database schema (tables, views, sequences, indexes), with unique, primary, foreign key and check constraints. - Export grants/privileges for users and groups. - Export range/list partitions and sub partitions. - Export a table selection (by specifying the table names). - Export Oracle schema to a PostgreSQL 8.4+ schema. - Export predefined functions, triggers, procedures, packages and package bodies. - Export full data or following a WHERE clause. - Full support of Oracle BLOB object as PG BYTEA. - Export Oracle views as PG tables. - Export Oracle user defined types. - Provide some basic automatic conversion of PLSQL code to PLPGSQL. - Works on any platform. - Export Oracle tables as foreign data wrapper tables. - Export materialized view. - Show a report of an Oracle database content. - Migration cost assessment of an Oracle database. - Migration difficulty level assessment of an Oracle database. - Migration cost assessment of PL/SQL code from a file. - Migration cost assessment of Oracle SQL queries stored in a file. - Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) - Export Oracle locator and spatial geometries into PostGis. - Export DBLINK as Oracle FDW. - Export SYNONYMS as views. - Export DIRECTORY as external table or directory for external_file extension. - Dispatch a list of SQL orders over multiple PostgreSQL connections - Perform a diff between Oracle and PostgreSQL database for test purpose. - MySQL/MariaDB and Microsoft SQL Server migration. Ora2Pg does its best to automatically convert your Oracle database to PostgreSQL but there's still manual works to do. The Oracle specific PL/SQL code generated for functions, procedures, packages and triggers has to be reviewed to match the PostgreSQL syntax. You will find some useful recommendations on porting Oracle PL/SQL code to PostgreSQL PL/PGSQL at "Converting from other Databases to PostgreSQL", section: Oracle (http://wiki.postgresql.org/wiki/Main_Page). See http://ora2pg.darold.net/report.html for a HTML sample of an Oracle database migration report. INSTALLATION All Perl modules can always be found at CPAN (http://search.cpan.org/). Just type the full name of the module (ex: DBD::Oracle) into the search input box, it will brings you the page for download. Releases of Ora2Pg stay at SF.net (https://sourceforge.net/projects/ora2pg/). Under Windows you should install Strawberry Perl (http://strawberryperl.com/) and the OSes corresponding Oracle clients. Since version 5.32 this Perl distribution include pre-compiled driver of DBD::Oracle and DBD::Pg. Requirement The Oracle Instant Client or a full Oracle installation must be installed on the system. You can download the RPM from Oracle download center: rpm -ivh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm or simply download the corresponding ZIP archives from Oracle download center and install them where you want, for example: /opt/oracle/instantclient_12_2/ You also need a modern Perl distribution (perl 5.10 and more). To connect to a database and proceed to his migration you need the DBI Perl module > 1.614. To migrate an Oracle database you need the DBD::Oracle Perl modules to be installed. To install DBD::Oracle and have it working you need to have the Oracle client libraries installed and the ORACLE_HOME environment variable must be defined. If you plan to export a MySQL database you need to install the Perl module DBD::MySQL which requires that the mysql client libraries are installed. If you plan to export a SQL Server database you need to install the Perl module DBD::ODBC which requires that the unixODBC package is installed. On some Perl distribution you may need to install the Time::HiRes Perl module. If your distribution doesn't include these Perl modules you can install them using CPAN: perl -MCPAN -e 'install DBD::Oracle' perl -MCPAN -e 'install DBD::MySQL' perl -MCPAN -e 'install DBD::ODBC' perl -MCPAN -e 'install Time::HiRes' otherwise use the packages provided by your distribution. Optional By default Ora2Pg dumps export to flat files, to load them into your PostgreSQL database you need the PostgreSQL client (psql). If you don't have it on the host running Ora2Pg you can always transfer these files to a host with the psql client installed. If you prefer to load export 'on the fly', the perl module DBD::Pg is required. Ora2Pg allows you to dump all output in a compressed gzip file, to do that you need the Compress::Zlib Perl module or if you prefer using bzip2 compression, the program bzip2 must be available in your PATH. If your distribution doesn't include these Perl modules you can install them using CPAN: perl -MCPAN -e 'install DBD::Pg' perl -MCPAN -e 'install Compress::Zlib' otherwise use the packages provided by your distribution. Instruction for SQL Server For SQL Server you need to install the unixodbc package and the Perl DBD::ODBC driver: sudo apt install unixodbc sudo apt install libdbd-odbc-perl or sudo yum install unixodbc sudo yum install perl-DBD-ODBC sudo yum install perl-DBD-Pg then install the Microsoft ODBC Driver for SQL Server. Follow the instructions relative to your operating system from here: https://docs.microsoft.com/fr-fr/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver16 Once it is done set the following in the /etc/odbcinst.ini file by adjusting the SQL Server ODBC driver version: [msodbcsql18] Description=Microsoft ODBC Driver 18 for SQL Server Driver=/opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.0.so.1.1 UsageCount=1 See ORACLE_DSN to know how to use the driver to connect to your MSSQL database. Installing Ora2Pg Like any other Perl Module Ora2Pg can be installed with the following commands: tar xjf ora2pg-x.x.tar.bz2 cd ora2pg-x.x/ perl Makefile.PL make && make install This will install Ora2Pg.pm into your site Perl repository, ora2pg into /usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. On Windows(tm) OSes you may use instead: perl Makefile.PL gmake && gmake install This will install scripts and libraries into your Perl site installation directory and the ora2pg.conf file as well as all documentation files into C:\ora2pg\ To install ora2pg in a different directory than the default one, simply use this command: perl Makefile.PL PREFIX= make && make install then set PERL5LIB to the path to your installation directory before using Ora2Pg. export PERL5LIB= ora2pg -c config/ora2pg.conf -t TABLE -b outdir/ Packaging If you want to build the binary package for your preferred Linux distribution take a look at the packaging/ directory of the source tarball. There is everything to build RPM, Slackware and Debian packages. See README file in that directory. Installing DBD::Oracle Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle database from perl DBI. To get DBD::Oracle get it from CPAN a perl module repository. After setting ORACLE_HOME and LD_LIBRARY_PATH environment variables as root user, install DBD::Oracle. Proceed as follow: export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib export ORACLE_HOME=/usr/lib/oracle/12.2/client64 perl -MCPAN -e 'install DBD::Oracle' If you are running for the first time it will ask many questions; you can keep defaults by pressing ENTER key, but you need to give one appropriate mirror site for CPAN to download the modules. Install through CPAN manually if the above doesn't work: #perl -MCPAN -e shell cpan> get DBD::Oracle cpan> quit cd ~/.cpan/build/DBD-Oracle* export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib export ORACLE_HOME=/usr/lib/oracle/11.2/client64 perl Makefile.PL make make install Installing DBD::Oracle require that the three Oracle packages: instant-client, SDK and SQLplus are installed as well as the libaio1 library. If you are using Instant Client from ZIP archives, the LD_LIBRARY_PATH and ORACLE_HOME will be the same and must be set to the directory where you have installed the files. For example: /opt/oracle/instantclient_12_2/ CONFIGURATION Ora2Pg configuration can be as simple as choosing the Oracle database to export and choose the export type. This can be done in a minute. By reading this documentation you will also be able to: - Select only certain tables and/or column for export. - Rename some tables and/or column during export. - Select data to export following a WHERE clause per table. - Delay database constraints during data loading. - Compress exported data to save disk space. - and much more. The full control of the Oracle database migration is taken though a single configuration file named ora2pg.conf. The format of this file consist in a directive name in upper case followed by tab character and a value. Comments are lines beginning with a #. There's no specific order to place the configuration directives, they are set at the time they are read in the configuration file. For configuration directives that just take a single value, you can use them multiple time in the configuration file but only the last occurrence found in the file will be used. For configuration directives that allow a list of value, you can use it multiple time, the values will be appended to the list. If you use the IMPORT directive to load a custom configuration file, directives defined in this file will be stores from the place the IMPORT directive is found, so it is better to put it at the end of the configuration file. Values set in command line options will override values from the configuration file. Ora2Pg usage First of all be sure that libraries and binaries path include the Oracle Instant Client installation: export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration file, if the file exist you can simply execute: /usr/local/bin/ora2pg or under Windows(tm) run ora2pg.bat file, located in your perl bin directory. Windows(tm) users may also find a template configuration file in C:\ora2pg If you want to call another configuration file, just give the path as command line argument: /usr/local/bin/ora2pg -c /etc/ora2pg/new_ora2pg.conf Here are all command line parameters available when using ora2pg: Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] -a | --allow str : Comma separated list of objects to allow from export. Can be used with SHOW_COLUMN too. -b | --basedir dir: Set the default output directory, where files resulting from exports will be stored. -c | --conf file : Set an alternate configuration file other than the default /etc/ora2pg/ora2pg.conf. -C | --cdc_file file: File used to store/read SCN per table during export. default: TABLES_SCN.log in the current directory. This is the file written by the --cdc_ready option. -d | --debug : Enable verbose output. -D | --data_type str : Allow custom type replacement at command line. -e | --exclude str: Comma separated list of objects to exclude from export. Can be used with SHOW_COLUMN too. -h | --help : Print this short help. -g | --grant_object type : Extract privilege from the given object type. See possible values with GRANT_OBJECT configuration. -i | --input file : File containing Oracle PL/SQL code to convert with no Oracle database connection initiated. -j | --jobs num : Number of parallel process to send data to PostgreSQL. -J | --copies num : Number of parallel connections to extract data from Oracle. -l | --log file : Set a log file. Default is stdout. -L | --limit num : Number of tuples extracted from Oracle and stored in memory before writing, default: 10000. -m | --mysql : Export a MySQL database instead of an Oracle schema. -M | --mssql : Export a Microsoft SQL Server database. -n | --namespace schema : Set the Oracle schema to extract from. -N | --pg_schema schema : Set PostgreSQL's search_path. -o | --out file : Set the path to the output file where SQL will be written. Default: output.sql in running directory. -O | --options : Used to override any configuration parameter, it can be used multiple time. Syntax: -O "PARAM_NAME=value" -p | --plsql : Enable PLSQL to PLPGSQL code conversion. -P | --parallel num: Number of parallel tables to extract at the same time. -q | --quiet : Disable progress bar. -r | --relative : use \ir instead of \i in the psql scripts generated. -s | --source DSN : Allow to set the Oracle DBI datasource. -S | --scn SCN : Allow to set the Oracle System Change Number (SCN) to use to export data. It will be used in the WHERE clause to get the data. It is used with action COPY or INSERT. -t | --type export: Set the export type. It will override the one given in the configuration file (TYPE). -T | --temp_dir dir: Set a distinct temporary directory when two or more ora2pg are run in parallel. -u | --user name : Set the Oracle database connection user. ORA2PG_USER environment variable can be used instead. -v | --version : Show Ora2Pg Version and exit. -w | --password pwd : Set the password of the Oracle database user. ORA2PG_PASSWD environment variable can be used instead. -W | --where clause : Set the WHERE clause to apply to the Oracle query to retrieve data. Can be used multiple time. --forceowner : Force ora2pg to set tables and sequences owner like in Oracle database. If the value is set to a username this one will be used as the objects owner. By default it's the user used to connect to the Pg database that will be the owner. --nls_lang code: Set the Oracle NLS_LANG client encoding. --client_encoding code: Set the PostgreSQL client encoding. --view_as_table str: Comma separated list of views to export as table. --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT --cost_unit_value minutes: Number of minutes for a cost evaluation unit. default: 5 minutes, corresponds to a migration conducted by a PostgreSQL expert. Set it to 10 if this is your first migration. --dump_as_html : Force ora2pg to dump report in HTML, used only with SHOW_REPORT. Default is to dump report as simple text. --dump_as_csv : As above but force ora2pg to dump report in CSV. --dump_as_json : As above but force ora2pg to dump report in JSON. --dump_as_sheet : Report migration assessment with one CSV line per database. --init_project name: Initialise a typical ora2pg project tree. Top directory will be created under project base dir. --project_base dir : Define the base dir for ora2pg project trees. Default is current directory. --print_header : Used with --dump_as_sheet to print the CSV header especially for the first run of ora2pg. --human_days_limit num : Set the number of person-days limit where the migration assessment level switch from B to C. Default is set to 5 person-days. --audit_user list : Comma separated list of usernames to filter queries in the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT and QUERY export type. --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. --pg_user name : Set the PostgreSQL user to use. --pg_pwd password : Set the PostgreSQL password to use. --count_rows : Force ora2pg to perform a real row count in TEST, TEST_COUNT and SHOW_TABLE actions. --no_header : Do not append Ora2Pg header to output file --oracle_speed : Use to know at which speed Oracle is able to send data. No data will be processed or written. --ora2pg_speed : Use to know at which speed Ora2Pg is able to send transformed data. Nothing will be written. --blob_to_lo : export BLOB as large objects, can only be used with action SHOW_COLUMN, TABLE and INSERT. --cdc_ready : use current SCN per table to export data and register them into a file named TABLES_SCN.log per default. It can be changed using -C | --cdc_file. --lo_import : use psql \lo_import command to import BLOB as large object. Can be use to import data with COPY and import large object manually in a second pass. It is recquired for BLOB > 1GB. See documentation for more explanation. --mview_as_table str: Comma separated list of materialized views to export as regular table. --drop_if_exists : Drop the object before creation if it exists. --delete clause : Set the DELETE clause to apply to the Oracle query to be applied before importing data. Can be used multiple time. --oracle_fdw_prefetch: Set the oracle_fdw prefetch value. Larger values generally result in faster data transfer at the cost of greater memory utilisation at the destination. See full documentation at https://ora2pg.darold.net/ for more help or see manpage with 'man ora2pg'. ora2pg will return 0 on success, 1 on error. It will return 2 when a child process has been interrupted and you've gotten the warning message: "WARNING: an error occurs during data export. Please check what's happen." Most of the time this is an OOM issue, first try reducing DATA_LIMIT value. For developers, it is possible to add your own custom option(s) in the Perl script ora2pg as any configuration directive from ora2pg.conf can be passed in lower case to the new Ora2Pg object instance. See ora2pg code on how to add your own option. Note that performance might be improved by updating stats on oracle: BEGIN DBMS_STATS.GATHER_SCHEMA_STATS DBMS_STATS.GATHER_DATABASE_STATS DBMS_STATS.GATHER_DICTIONARY_STATS END; Generate a migration template The two options --project_base and --init_project when used indicate to ora2pg that he has to create a project template with a work tree, a configuration file and a script to export all objects from the Oracle database. Here a sample of the command usage: ora2pg --project_base /app/migration/ --init_project test_project Creating project test_project. /app/migration/test_project/ schema/ dblinks/ directories/ functions/ grants/ mviews/ packages/ partitions/ procedures/ sequences/ synonyms/ tables/ tablespaces/ triggers/ types/ views/ sources/ functions/ mviews/ packages/ partitions/ procedures/ triggers/ types/ views/ data/ config/ reports/ Generating generic configuration file Creating script export_schema.sh to automate all exports. Creating script import_all.sh to automate all imports. It create a generic config file where you just have to define the Oracle database connection and a shell script called export_schema.sh. The sources/ directory will contains the Oracle code, the schema/ will contains the code ported to PostgreSQL. The reports/ directory will contains the html reports with the migration cost assessment. If you want to use your own default config file, use the -c option to give the path to that file. Rename it with .dist suffix if you want ora2pg to apply the generic configuration values otherwise, the configuration file will be copied untouched. Once you have set the connection to the Oracle Database you can execute the script export_schema.sh that will export all object type from your Oracle database and output DDL files into the schema's subdirectories. At end of the export it will give you the command to export data later when the import of the schema will be done and verified. You can choose to load the DDL files generated manually or use the second script import_all.sh to import those file interactively. If this kind of migration is not something current for you it's recommended you to use those scripts. Oracle database connection There's 5 configuration directives to control the access to the Oracle database. ORACLE_HOME Used to set ORACLE_HOME environment variable to the Oracle libraries required by the DBD::Oracle Perl module. ORACLE_DSN This directive is used to set the data source name in the form standard DBI DSN. For example: dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 or dbi:Oracle:DB_SID On 18c this could be for example: dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 for the second notation the SID should be declared in the well known file $ORACLE_HOME/network/admin/tnsnames.ora or in the path given to the TNS_ADMIN environment variable. For MySQL the DSN will lool like this: dbi:mysql:host=192.168.1.10;database=sakila;port=3306 the 'sid' part is replaced by 'database'. For MS SQL Server it will look like this: dbi:ODBC:driver=msodbcsql18;server=mydb.database.windows.net;database=testdb;TrustServerCertificate=yes ORACLE_USER et ORACLE_PWD These two directives are used to define the user and password for the Oracle database connection. Note that if you can it is better to login as Oracle super admin to avoid grants problem during the database scan and be sure that nothing is missing. If you do not supply a credential with ORACLE_PWD and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If ORACLE_USER is not set it will be asked interactively too. To connect to a local ORACLE instance with connections "as sysdba" you have to set ORACLE_USER to "/" and an empty password. To make a connection using an Oracle Secure External Password Store (SEPS), first configure the Oracle Wallet and then set both the ORACLE_USER and ORACLE_PWD directives to the special value of "__SEPS__" (without the quotes but with the double underscore). USER_GRANTS Set this directive to 1 if you connect the Oracle database as simple user and do not have enough grants to extract things from the DBA_... tables. It will use tables ALL_... instead. Warning: if you use export type GRANT, you must set this configuration option to 0 or it will not work. TRANSACTION This directive may be used if you want to change the default isolation level of the data export transaction. Default is now to set the level to a serializable transaction to ensure data consistency. The allowed values for this directive are: readonly: 'SET TRANSACTION READ ONLY', readwrite: 'SET TRANSACTION READ WRITE', serializable: 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' committed: 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED', Releases before 6.2 used to set the isolation level to READ ONLY transaction but in some case this was breaking data consistency so now default is set to SERIALIZABLE. INPUT_FILE This directive did not control the Oracle database connection or unless it purely disables the use of any Oracle database by accepting a file as argument. Set this directive to a file containing PL/SQL Oracle Code like function, procedure or full package body to prevent Ora2Pg from connecting to an Oracle database and just apply his conversion tool to the content of the file. This can be used with the most of export types: TABLE, TRIGGER, PROCEDURE, VIEW, FUNCTION or PACKAGE, etc. ORA_INITIAL_COMMAND This directive can be used to send an initial command to Oracle, just after the connection. For example to unlock a policy before reading objects or to set some session parameters. This directive can be used multiple times. Data encryption with Oracle server If your Oracle Client config file already includes the encryption method, then DBD:Oracle uses those settings to encrypt the connection while you extract the data. For example if you have configured the Oracle Client config file (sqlnet.ora or .sqlnet) with the following information: # Configure encryption of connections to Oracle SQLNET.ENCRYPTION_CLIENT = required SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) SQLNET.CRYPTO_SEED = 'should be 10-70 random characters' Any tool that uses the Oracle client to talk to the database will be encrypted if you setup session encryption like above. For example, Perl's DBI uses DBD-Oracle, which uses the Oracle client for actually handling database communication. If the installation of Oracle client used by Perl is setup to request encrypted connections, then your Perl connection to an Oracle database will also be encrypted. Full details at https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 Testing connection Once you have set the Oracle database DSN you can execute ora2pg to see if it works: ora2pg -t SHOW_VERSION -c config/ora2pg.conf will show the Oracle database server version. Take some time here to test your installation as most problems take place here, the other configuration steps are more technical. Troubleshooting If the output.sql file has not exported anything other than the Pg transaction header and footer there's two possible reasons. The perl script ora2pg dump an ORA-XXX error, that mean that your DSN or login information are wrong, check the error and your settings and try again. The perl script says nothing and the output file is empty: the user lacks permission to extract something from the database. Try to connect to Oracle as super user or take a look at directive USER_GRANTS above and at next section, especially the SCHEMA directive. LOGFILE By default all messages are sent to the standard output. If you give a file path to that directive, all output will be appended to this file. Oracle schema to export The Oracle database export can be limited to a specific Schema or Namespace, this can be mandatory following the database connection user. SCHEMA This directive is used to set the schema name to use during export. For example: SCHEMA APPS will extract objects associated to the APPS schema. When no schema name is provided and EXPORT_SCHEMA is enabled, Ora2Pg will export all objects from all schema of the Oracle instance with their names prefixed with the schema name. EXPORT_SCHEMA By default the Oracle schema is not exported into the PostgreSQL database and all objects are created under the default Pg namespace. If you want to also export this schema and create all objects under this namespace, set the EXPORT_SCHEMA directive to 1. This will set the schema search_path at top of export SQL file to the schema name set in the SCHEMA directive with the default pg_catalog schema. If you want to change this path, use the directive PG_SCHEMA. CREATE_SCHEMA Enable/disable the CREATE SCHEMA SQL order at starting of the output file. It is enable by default and concern on TABLE export type. COMPILE_SCHEMA By default Ora2Pg will only export valid PL/SQL code. You can force Oracle to compile again the invalidated code to get a chance to have it obtain the valid status and then be able to export it. Enable this directive to force Oracle to compile schema before exporting code. When this directive is enabled and SCHEMA is set to a specific schema name, only invalid objects in this schema will be recompiled. If SCHEMA is not set then all schema will be recompiled. To force recompile invalid object in a specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. This will ask to Oracle to validate the PL/SQL that could have been invalidate after a export/import for example. The 'VALID' or 'INVALID' status applies to functions, procedures, packages and user defined types. It also concern disabled triggers. EXPORT_INVALID If the above configuration directive is not enough to validate your PL/SQL code enable this configuration directive to allow export of all PL/SQL code even if it is marked as invalid. The 'VALID' or 'INVALID' status applies to functions, procedures, packages, triggers and user defined types. PG_SCHEMA Allow you to defined/force the PostgreSQL schema to use. By default if you set EXPORT_SCHEMA to 1 the PostgreSQL search_path will be set to the schema name exported set as value of the SCHEMA directive. The value can be a comma delimited list of schema name but not when using TABLE export type because in this case it will generate the CREATE SCHEMA statement and it doesn't support multiple schema name. For example, if you set PG_SCHEMA to something like "user_schema, public", the search path will be set like this: SET search_path = user_schema, public; forcing the use of an other schema (here user_schema) than the one from Oracle schema set in the SCHEMA directive. You can also set the default search_path for the PostgreSQL user you are using to connect to the destination database by using: ALTER ROLE username SET search_path TO user_schema, public; in this case you don't have to set PG_SCHEMA. SYSUSERS Without explicit schema, Ora2Pg will export all objects that not belongs to system schema or role: SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER Following your Oracle installation you may have several other system role defined. To append these users to the schema exclusion list, just set the SYSUSERS configuration directive to a comma-separated list of system user to exclude. For example: SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH will add users INTERNAL and SYSDBA to the schema exclusion list. FORCE_OWNER By default the owner of the database objects is the one you're using to connect to PostgreSQL using the psql command. If you use an other user (postgres for example) you can force Ora2Pg to set the object owner to be the one used in the Oracle database by setting the directive to 1, or to a completely different username by setting the directive value to that username. FORCE_SECURITY_INVOKER Ora2Pg use the function's security privileges set in Oracle and it is often defined as SECURITY DEFINER. If you want to override those security privileges for all functions and use SECURITY DEFINER instead, enable this directive. USE_TABLESPACE When enabled this directive force ora2pg to export all tables, indexes constraint and indexes using the tablespace name defined in Oracle database. This works only with tablespace that are not TEMP, USERS and SYSTEM. WITH_OID Activating this directive will force Ora2Pg to add WITH (OIDS) when creating tables or views as tables. Default is same as PostgreSQL, disabled. LOOK_FORWARD_FUNCTION List of schema to get functions/procedures meta information that are used in the current schema export. When replacing call to function with OUT parameters, if a function is declared in an other package then the function call rewriting can not be done because Ora2Pg only knows about functions declared in the current schema. By setting a comma separated list of schema as value of this directive, Ora2Pg will look forward in these packages for all functions/procedures/packages declaration before proceeding to current schema export. NO_FUNCTION_METADATA Force Ora2Pg to not look for function declaration. Note that this will prevent Ora2Pg to rewrite function replacement call if needed. Do not enable it unless looking forward at function breaks other export. Export type The export action is perform following a single configuration directive 'TYPE', some other add more control on what should be really exported. TYPE Here are the different values of the TYPE directive, default is TABLE: - TABLE: Extract all tables with indexes, primary keys, unique keys, foreign keys and check constraints. - VIEW: Extract only views. - GRANT: Extract roles converted to Pg groups, users and grants on all objects. - SEQUENCE: Extract all sequence and their last position. - TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). - TRIGGER: Extract triggers defined following actions. - FUNCTION: Extract functions. - PROCEDURE: Extract procedures. - PACKAGE: Extract packages and package bodies. - INSERT: Extract data as INSERT statement. - COPY: Extract data as COPY statement. - PARTITION: Extract range and list Oracle partitions with subpartitions. - TYPE: Extract user defined Oracle type. - FDW: Export Oracle tables as foreign table for Oracle, MySQL and SQL Server FDW. - MVIEW: Export materialized view. - QUERY: Try to automatically convert Oracle SQL queries. - KETTLE: Generate XML ktr template files to be used by Kettle. - DBLINK: Generate oracle foreign data wrapper server to use as dblink. - SYNONYM: Export Oracle's synonyms as views on other schema's objects. - DIRECTORY: Export Oracle's directories as external_file extension objects. - LOAD: Dispatch a list of queries over multiple PostgreSQl connections. - TEST: perform a diff between Oracle and PostgreSQL database. - TEST_COUNT: perform a row count diff between Oracle and PostgreSQL table. - TEST_VIEW: perform a count on both side of number of rows returned by views. - TEST_DATA: perform data validation check on rows at both sides. - SEQUENCE_VALUES: export DDL to set the last values of sequences Only one type of export can be perform at the same time so the TYPE directive must be unique. If you have more than one only the last found in the file will be registered. Some export type can not or should not be load directly into the PostgreSQL database and still require little manual editing. This is the case for GRANT, TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, QUERY and PACKAGE export types especially if you have PLSQL code or Oracle specific SQL in it. For TABLESPACE you must ensure that file path exist on the system and for SYNONYM you may ensure that the object's owners and schemas correspond to the new PostgreSQL database design. Note that you can chained multiple export by giving to the TYPE directive a comma-separated list of export type, but in this case you must not use COPY or INSERT with other export type. Ora2Pg will convert Oracle partition using table inheritance, trigger and functions. See document at Pg site: http://www.postgresql.org/docs/current/interactive/ddl-partitioning. html The TYPE export allow export of user defined Oracle type. If you don't use the --plsql command line parameter it simply dump Oracle user type asis else Ora2Pg will try to convert it to PostgreSQL syntax. The KETTLE export type requires that the Oracle and PostgreSQL DNS are defined. Since Ora2Pg v8.1 there's three new export types: SHOW_VERSION : display Oracle version SHOW_SCHEMA : display the list of schema available in the database. SHOW_TABLE : display the list of tables available. SHOW_COLUMN : display the list of tables columns available and the Ora2PG conversion type from Oracle to PostgreSQL that will be applied. It will also warn you if there's PostgreSQL reserved words in Oracle object names. Here is an example of the SHOW_COLUMN output: [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: 'CURRENT_SCHEMA' is a reserved word in PostgreSQL) CONSTRAINT : NUMBER(22) => bigint (Warning: 'CONSTRAINT' is a reserved word in PostgreSQL) FREEZE : VARCHAR2(25) => varchar(25) (Warning: 'FREEZE' is a reserved word in PostgreSQL) ... [6] TABLE LOCATIONS (23 rows) LOCATION_ID : NUMBER(4) => smallint STREET_ADDRESS : VARCHAR2(40) => varchar(40) POSTAL_CODE : VARCHAR2(12) => varchar(12) CITY : VARCHAR2(30) => varchar(30) STATE_PROVINCE : VARCHAR2(25) => varchar(25) COUNTRY_ID : CHAR(2) => char(2) Those extraction keywords are use to only display the requested information and exit. This allows you to quickly know on what you are going to work. The SHOW_COLUMN allow an other ora2pg command line option: '--allow relname' or '-a relname' to limit the displayed information to the given table. The SHOW_ENCODING export type will display the NLS_LANG and CLIENT_ENCODING values that Ora2Pg will used and the real encoding of the Oracle database with the corresponding client encoding that could be used with PostgreSQL Ora2Pg allow you to export your Oracle, MySQL or MSSQL table definition to be use with the oracle_fdw, mysql_fdw or tds_fdw foreign data wrapper. By using type FDW your tables will be exported as follow: CREATE FOREIGN TABLE oratab ( id integer NOT NULL, text character varying(30), floating double precision NOT NULL ) SERVER oradb OPTIONS (table 'ORATAB'); Now you can use the table like a regular PostgreSQL table. Release 10 adds a new export type destined to evaluate the content of the database to migrate, in terms of objects and cost to end the migration: SHOW_REPORT : show a detailed report of the Oracle database content. Here is a sample of report: http://ora2pg.darold.net/report.html There also a more advanced report with migration cost. See the dedicated chapter about Migration Cost Evaluation. ESTIMATE_COST Activate the migration cost evaluation. Must only be used with SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE and QUERY export type. Default is disabled. You may want to use the --estimate_cost command line option instead to activate this functionality. Note that enabling this directive will force PLSQL_PGSQL activation. COST_UNIT_VALUE Set the value in minutes of the migration cost evaluation unit. Default is five minutes per unit. See --cost_unit_value to change the unit value at command line. DUMP_AS_HTML By default when using SHOW_REPORT the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in HTML format. See http://ora2pg.darold.net/report.html for a sample report. HUMAN_DAYS_LIMIT Use this directive to redefined the number of person-days limit where the migration assessment level must switch from B to C. Default is set to 10 person-days. JOBS This configuration directive adds multiprocess support to COPY, FUNCTION and PROCEDURE export type, the value is the number of process to use. Default is multiprocess disable. This directive is used to set the number of cores to used to parallelize data import into PostgreSQL. During FUNCTION or PROCEDURE export type each function will be translated to plpgsql using a new process, the performances gain can be very important when you have tons of function to convert. There's no limitation in parallel processing than the number of cores and the PostgreSQL I/O performance capabilities. Doesn't work under Windows Operating System, it is simply disabled. ORACLE_COPIES This configuration directive adds multiprocess support to extract data from Oracle. The value is the number of process to use to parallelize the select query. Default is parallel query disable. The parallelism is built on splitting the query following of the number of cores given as value to ORACLE_COPIES as follow: SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC where COLUMN is a technical key like a primary or unique key where split will be based and the current core used by the query (CUR_PROC). You can also force the column name to use using the DEFINED_PK configuration directive. Doesn't work under Windows Operating System, it is simply disabled. DEFINED_PK This directive is used to defined the technical key to used to split the query between number of cores set with the ORACLE_COPIES variable. For example: DEFINED_PK EMPLOYEES:employee_id The parallel query that will be used supposing that -J or ORACLE_COPIES is set to 8: SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N where N is the current process forked starting from 0. PARALLEL_TABLES This directive is used to defined the number of tables that will be processed in parallel for data extraction. The limit is the number of cores on your machine. Ora2Pg will open one database connection for each parallel table extraction. This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS, so the real number of process that will be used is PARALLEL_TABLES * JOBS. Note that this directive when set upper that 1 will also automatically enable the FILE_PER_TABLE directive if your are exporting to files. This is used to export tables and views in separate files. Use PARALLEL_TABLES to use parallelism with COPY, INSERT and TEST_DATA actions. It is also useful with TEST, TEST_COUNT, and SHOW_TABLE if --count_rows is used for real row count. DEFAULT_PARALLELISM_DEGREE You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in each query used to export data from Oracle by setting a value upper than 1 to this directive. A value of 0 or 1 disable the use of parallel hint. Default is disabled. FDW_SERVER This directive is used to set the name of the foreign data server that is used in the "CREATE SERVER name FOREIGN DATA WRAPPER ..." command. This name will then be used in the "CREATE FOREIGN TABLE ..." SQL commands and to import data using oracle_fdw. Default is no foreign server defined. This only concerns export type FDW, COPY and INSERT. For export type FDW the default value is orcl. FDW_IMPORT_SCHEMA Schema where foreign tables for data migration will be created. If you use several instances of ora2pg for data migration through the foreign data wrapper, you might need to change the name of the schema for each instance. Default: ora2pg_fdw_import ORACLE_FDW_PREFETCH The default behaviour of Ora2Pg is to NOT set the "prefetch" option for oracle_fdw when used for COPY and INSERT. This directive allows the prefetch to be set. See oracle_fdw documentation for the current default. ORACLE_FDW_COPY_MODE When using Ora2Pg COPY with oracle_fdw it is possible to use two different modes: 1) "local", which uses psql on the host running Ora2Pg for the "TO" binary stream; 2) "server", which uses PostgreSQL server-side COPY for the "TO" binary stream. Both modes use psql for the "FROM STDIN BINARY". However, "local" runs the psql "FROM STDIN BINARY" on host Ora2Pg is run from, whereas "server" runs the psql "FROM STDIN BINARY" on the PostgreSQL server. "local" mode should work on any PostgreSQL-based system, including managed offerings, which are not expected to support use of "server" mode due to permissions. The default is "local" as this is compatible with more configurations. ORACLE_FDW_COPY_FORMAT When using Ora2Pg COPY with oracle_fdw it is possible to use either BINARY or CSV data format. BINARY provides better performance, however, requires exact data type matching between the FDW and destination table. CSV provides greater flexibiliity with respect to data type matching: if the FDW and destination data types are functionally-compatible the columns can be copied. The default is "binary". DROP_FOREIGN_SCHEMA By default Ora2Pg drops the temporary schema ora2pg_fdw_import used to import the Oracle foreign schema before each new import. If you want to preserve the existing schema because of modifications or the use of a third party server, disable this directive. EXTERNAL_TO_FDW This directive, enabled by default, allow to export Oracle's External Tables as file_fdw foreign tables. To not export these tables at all, set the directive to 0. INTERNAL_DATE_MAX Internal timestamp retrieves from custom type are extracted in the following format: 01-JAN-77 12.00.00.000000 AM. It is impossible to know the exact century that must be used, so by default any year below 49 will be added to 2000 and others to 1900. You can use this directive to change the default value 49. this is only relevant if you have user defined type with a column timestamp. AUDIT_USER Set the comma separated list of username that must be used to filter queries from the DBA_AUDIT_TRAIL table. Default is to not scan this table and to never look for queries. This parameter is used only with SHOW_REPORT and QUERY export type with no input file for queries. Note that queries will be normalized before output unlike when a file is given at input using the -i option or INPUT directive. FUNCTION_CHECK Disable this directive if you want to disable check_function_bodies. SET check_function_bodies = false; It disables validation of the function body string during CREATE FUNCTION. Default is to use de postgresql.conf setting that enable it by default. ENABLE_BLOB_EXPORT Exporting BLOB takes time, in some circumstances you may want to export all data except the BLOB columns. In this case disable this directive and the BLOB columns will not be included into data export. Take care that the target bytea column do not have a NOT NULL constraint. ENABLE_CLOB_EXPORT Same behavior as ENABLE_BLOB_EXPORT but for CLOB. DATA_EXPORT_ORDER By default data export order will be done by sorting on table name. If you have huge tables at end of alphabetic order and you are using multiprocess, it can be better to set the sort order on size so that multiple small tables can be processed before the largest tables finish. In this case set this directive to size. Possible values are name and size. Note that export type SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or INSERT export type. If you want to give you custom export order, just give a filename as value that contains the ordered list of table to export. Must be a list of one table per line, in uppercase for Oracle. Limiting objects to export You may want to export only a part of an Oracle database, here are a set of configuration directives that will allow you to control what parts of the database should be exported. ALLOW This directive allows you to set a list of objects on which the export must be limited, excluding all other objects in the same type of export. The value is a space or comma-separated list of objects name to export. You can include valid regex into the list. For example: ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ will export objects with name EMPLOYEES, COUNTRIES, all objects beginning with 'SALE_' and all objects with a name ending by '_GEOM_SEQ'. The object depends of the export type. Note that regex will not works with 8i database, you must use the % placeholder instead, Ora2Pg will use the LIKE operator. This is the manner to declare global filters that will be used with the current export type. You can also use extended filters that will be applied on specific objects or only on their related export type. For example: ora2pg -p -c ora2pg.conf -t TRIGGER -a 'TABLE[employees]' will limit export of trigger to those defined on table employees. If you want to extract all triggers but not some INSTEAD OF triggers: ora2pg -c ora2pg.conf -t TRIGGER -e 'VIEW[trg_view_.*]' Or a more complex form: ora2pg -p -c ora2pg.conf -t TABLE -a 'TABLE[EMPLOYEES]' \ -e 'INDEX[emp_.*];CKEY[emp_salary_min]' This command will export the definition of the employee table but will exclude all index beginning with 'emp_' and the CHECK constraint called 'emp_salary_min'. When exporting partition you can exclude some partition tables by using ora2pg -p -c ora2pg.conf -t PARTITION -e 'PARTITION[PART_199.* PART_198.*]' This will exclude partitioned tables for year 1980 to 1999 from the export but not the main partition table. The trigger will also be adapted to exclude those table. With GRANT export you can use this extended form to exclude some users from the export or limit the export to some others: ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' or ora2pg -p -c ora2pg.conf -t GRANT -a 'GRANT[USER1 USER2]' will limit export grants to users USER1 and USER2. But if you don't want to export grants on some functions for these users, for example: ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' -e 'FUNCTION[adm_.*];PROCEDURE[adm_.*]' Advanced filters may need some learning. Oracle doesn't allow the use of lookahead expression so you may want to exclude some object that match the ALLOW regexp you have defined. For example if you want to export all table starting with E but not those starting with EXP it is not possible to do that in a single expression. This is why you can start a regular expression with the ! character to exclude object matching the regexp given just after. Our previous example can be written as follow: ALLOW E.* !EXP.* it will be translated into: REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') in the object search expression. EXCLUDE This directive is the opposite of the previous, it allow you to define a space or comma-separated list of object name to exclude from the export. You can include valid regex into the list. For example: EXCLUDE EMPLOYEES TMP_.* COUNTRIES will exclude object with name EMPLOYEES, COUNTRIES and all tables beginning with 'tmp_'. For example, you can ban from export some unwanted function with this directive: EXCLUDE write_to_.* send_mail_.* this example will exclude all functions, procedures or functions in a package with the name beginning with those regex. Note that regex will not work with 8i database, you must use the % placeholder instead, Ora2Pg will use the NOT LIKE operator. See above (directive 'ALLOW') for the extended syntax. NO_EXCLUDED_TABLE By default Ora2Pg exclude from export some Oracle "garbage" tables that should never be part of an export. This behavior generates a lot of REGEXP_LIKE expressions which are slowing down the export when looking at tables. To disable this behavior enable this directive, you will have to exclude or clean up later by yourself the unwanted tables. The regexp used to exclude the table are defined in the array @EXCLUDED_TABLES in lib/Ora2Pg.pm. Note this is behavior is independant to the EXCLUDE configuration directive. VIEW_AS_TABLE Set which view to export as table. By default none. Value must be a list of view name or regexp separated by space or comma. If the object name is a view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. See chapter "Exporting views as PostgreSQL table" for more details. MVIEW_AS_TABLE Set which materialized view to export as table. By default none. Value must be a list of materialized view name or regexp separated by space or comma. If the object name is a materialized view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. NO_VIEW_ORDERING By default Ora2Pg try to order views to avoid error at import time with nested views. With a huge number of views this can take a very long time, you can bypass this ordering by enabling this directive. GRANT_OBJECT When exporting GRANTs you can specify a comma separated list of objects for which privilege will be exported. Default is export for all objects. Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY. Only one object type is allowed at a time. For example set it to TABLE if you just want to export privilege on tables. You can use the -g option to overwrite it. When used this directive prevent the export of users unless it is set to USER. In this case only users definitions are exported. WHERE This directive allows you to specify a WHERE clause filter when dumping the contents of tables. Value is constructs as follows: TABLE_NAME[WHERE_CLAUSE], or if you have only one where clause for each table just put the where clause as the value. Both are possible too. Here are some examples: # Global where clause applying to all tables included in the export WHERE 1=1 # Apply the where clause only on table TABLE_NAME WHERE TABLE_NAME[ID1='001'] # Applies two different clause on tables TABLE_NAME and OTHER_TABLE # and a generic where clause on DATE_CREATE to all other tables WHERE TABLE_NAME[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' OTHER_TABLE[NAME='test'] Any where clause not included into a table name bracket clause will be applied to all exported table including the tables defined in the where clause. These WHERE clauses are very useful if you want to archive some data or at the opposite only export some recent data. To be able to quickly test data import it is useful to limit data export to the first thousand tuples of each table. For Oracle define the following clause: WHERE ROWNUM < 1000 and for MySQL, use the following: WHERE 1=1 LIMIT 1,1000 This can also be restricted to some tables data export. Command line option -W or --where will override this directive for the global part and per table if the table names is the same. TOP_MAX This directive is used to limit the number of item shown in the top N lists like the top list of tables per number of rows and the top list of largest tables in megabytes. By default it is set to 10 items. LOG_ON_ERROR Enable this directive if you want to continue direct data import on error. When Ora2Pg received an error in the COPY or INSERT statement from PostgreSQL it will log the statement to a file called TABLENAME_error.log in the output directory and continue to next bulk of data. Like this you can try to fix the statement and manually reload the error log file. Default is disabled: abort import on error. REPLACE_QUERY Sometime you may want to extract data from an Oracle table but you need a custom query for that. Not just a "SELECT * FROM table" like Ora2Pg do but a more complex query. This directive allows you to overwrite the query used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. If you have multiple table to extract by replacing the Ora2Pg query, you can define multiple REPLACE_QUERY lines. REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] Control of Full Text Search export Several directives can be used to control the way Ora2Pg will export the Oracle's Text search indexes. By default CONTEXT indexes will be exported to PostgreSQL FTS indexes but CTXCAT indexes will be exported as indexes using the pg_trgm extension. CONTEXT_AS_TRGM Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes and CTXCAT indexes using pg_trgm. Most of the time using pg_trgm is enough, this is why this directive stand for. You need to create the pg_trgm extension into the destination database before importing the objects: CREATE EXTENSION pg_trgm; FTS_INDEX_ONLY By default Ora2Pg creates a function-based index to translate Oracle Text indexes. CREATE INDEX ON t_document USING gin(to_tsvector('pg_catalog.french', title)); You will have to rewrite the CONTAIN() clause using to_tsvector(), example: SELECT id,title FROM t_document WHERE to_tsvector(title)) @@ to_tsquery('search_word'); To force Ora2Pg to create an extra tsvector column with a dedicated triggers for FTS indexes, disable this directive. In this case, Ora2Pg will add the column as follow: ALTER TABLE t_document ADD COLUMN tsv_title tsvector; Then update the column to compute FTS vectors if data have been loaded before UPDATE t_document SET tsv_title = to_tsvector('pg_catalog.french', coalesce(title,'')); To automatically update the column when a modification in the title column appears, Ora2Pg adds the following trigger: CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ BEGIN IF TG_OP = 'INSERT' OR new.title != old.title THEN new.tsv_title := to_tsvector('pg_catalog.french', coalesce(new.title,'')); END IF; return new; END $$ LANGUAGE plpgsql; CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE ON t_document FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); When the Oracle text index is defined over multiple column, Ora2Pg will use setweight() to set a weight in the order of the column declaration. FTS_CONFIG Use this directive to force text search configuration to use. When it is not set, Ora2Pg will autodetect the stemmer used by Oracle for each index and pg_catalog.english if the information is not found. USE_UNACCENT If you want to perform your text search in an accent insensitive way, enable this directive. Ora2Pg will create an helper function over unaccent() and creates the pg_trgm indexes using this function. With FTS Ora2Pg will redefine your text search configuration, for example: CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; then set the FTS_CONFIG ora2pg.conf directive to fr instead of pg_catalog.english. When enabled, Ora2pg will create the wrapper function: CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$ SELECT public.unaccent('public.unaccent', $1); $$ LANGUAGE sql IMMUTABLE COST 1; the indexes are exported as follow: CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document USING gin (unaccent_immutable(title) gin_trgm_ops); In your queries you will need to use the same function in the search to be able to use the function-based index. Example: SELECT * FROM t_document WHERE unaccent_immutable(title) LIKE '%donnees%'; USE_LOWER_UNACCENT Same as above but call lower() in the unaccent_immutable() function: CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$ SELECT lower(public.unaccent('public.unaccent', $1)); $$ LANGUAGE sql IMMUTABLE; Modifying object structure One of the great usage of Ora2Pg is its flexibility to replicate Oracle database into PostgreSQL database with a different structure or schema. There's three configuration directives that allow you to map those differences. REORDERING_COLUMNS Enable this directive to reordering columns and minimized the footprint on disc, so that more rows fit on a data page, which is the most important factor for speed. Default is disabled, that mean the same order than in Oracle tables definition, that's should be enough for most usage. This directive is only used with TABLE export. MODIFY_STRUCT This directive allows you to limit the columns to extract for a given table. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... for example: MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) This will only extract columns 'id' and 'dossier' from table T_TEST1 and columns 'id' and 'fichier' from the T_TEST2 table. This directive can only be used with TABLE, COPY or INSERT export. With TABLE export create table DDL will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. EXCLUDE_COLUMNS Instead of redefining the table structure with MODIFY_STRUCT you may want to exclude some columns from the table export. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: EXCLUDE_COLUMNS NOM_TABLE(nomcol1,nomcol2,...) ... for example: EXCLUDE_COLUMNS T_TEST1(id,dossier) T_TEST2(id,fichier) This will exclude from the export columns 'id' and 'dossier' from table T_TEST1 and columns 'id' and 'fichier' from the T_TEST2 table. This directive can only be used with TABLE, COPY or INSERT export. With TABLE export create table DDL will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. REPLACE_TABLES This directive allows you to remap a list of Oracle table name to a PostgreSQL table name during export. The value is a list of space-separated values with the following structure: REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 Oracle tables ORIG_TBNAME1 and ORIG_TBNAME2 will be respectively renamed into DEST_TBNAME1 and DEST_TBNAME2 REPLACE_COLS Like table name, the name of the column can be remapped to a different name using the following syntax: REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) For example: REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) will rename Oracle columns 'dico' and 'dossier' from table T_TEST into new name 'dictionary' and 'folder'. REPLACE_AS_BOOLEAN If you want to change the type of some Oracle columns into PostgreSQL boolean during the export you can define here a list of tables and column separated by space as follow. REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 The values set in the boolean columns list will be replaced with the 't' and 'f' following the default replacement values and those additionally set in directive BOOLEAN_VALUES. Note that if you have modified the table name with REPLACE_TABLES and/or the column's name, you need to use the name of the original table and/or column. REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 You can also give a type and a precision to automatically convert all fields of that type as a boolean. For example: REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 will also replace any field of type number(1) or char(1) as a boolean in all exported tables. BOOLEAN_VALUES Use this to add additional definition of the possible boolean values used in Oracle fields. You must set a space-separated list of TRUE:FALSE values. By default here are the values recognized by Ora2Pg: BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled Any values defined here will be added to the default list. REPLACE_ZERO_DATE When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL. This could be a problem if your column is defined with NOT NULL constraint. If you can not remove the constraint, use this directive to set an arbitral date that will be used instead. You can also use -INFINITY if you don't want to use a fake date. INDEXES_SUFFIX Add the given value as suffix to indexes names. Useful if you have indexes with same name as tables. For example: INDEXES_SUFFIX _idx will add _idx at ed of all index name. Not so common but can help. INDEXES_RENAMING Enable this directive to rename all indexes using tablename_columns_names. Could be very useful for database that have multiple time the same index name or that use the same name than a table, which is not allowed by PostgreSQL Disabled by default. USE_INDEX_OPCLASS Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B-tree indexes on the corresponding types. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (LIKE or POSIX regular expressions) when the database does not use the standard "C" locale. If you enable, with value 1, this will force Ora2Pg to export all indexes defined on varchar2() and char() columns using those operators. If you set it to a value greater than 1 it will only change indexes on columns where the character limit is greater or equal than this value. For example, set it to 128 to create these kind of indexes on columns of type varchar2(N) where N >= 128. RENAME_PARTITION Enable this directive if you want that your partition tables will be renamed. Disabled by default. If you have multiple partitioned table, when exported to PostgreSQL some partitions could have the same name but different parent tables. This is not allowed, table name must be unique, in this case enable this directive. A partition will be renamed following the rule: "tablename"_part"pos" where "pos" is the partition number. For subpartition this is: "tablename"_part"pos"_subpart"pos" If this is partition/subpartition default: "tablename"_part_default "tablename"_part"pos"_subpart_default DISABLE_PARTITION If you don't want to reproduce the partitioning like in Oracle and want to export all partitioned Oracle data into the main single table in PostgreSQL enable this directive. Ora2Pg will export all data into the main table name. Default is to use partitioning, Ora2Pg will export data from each partition and import them into the PostgreSQL dedicated partition table. PARTITION_BY_REFERENCE How to export partition by reference. Possible values are none, duplicate or the number of hash partition to create. Default is none to not export the partitions by reference. Value none mean no translation and export of partition by reference like before. Value 'duplicate' will duplicate the referenced column in the partitioned table and apply the same partitioning from the referenced table to the partitioned table. If the value is a number, the table will be partitioned with the HASH method using the value as the modulo. For example if you set it to 4 it will create 4 HASH partitions. DISABLE_UNLOGGED By default Ora2Pg export Oracle tables with the NOLOGGING attribute as UNLOGGED tables. You may want to fully disable this feature because you will lose all data from unlogged tables in case of a PostgreSQL crash. Set it to 1 to export all tables as normal tables. DOUBLE_MAX_VARCHAR Increase varchar max character constraints to support PostgreSQL two bytes character encoding when the source database applies the length constraint on characters not bytes. Default disabled. Oracle Spatial to PostGis Ora2Pg fully export Spatial object from Oracle database. There's some configuration directives that could be used to control the export. AUTODETECT_SPATIAL_TYPE By default Ora2Pg is looking at indexes to see the spatial constraint type and dimensions defined under Oracle. Those constraints are passed as at index creation using for example: CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS('sdo_indx_dims=2, layer_gtype=point'); If those Oracle constraints parameters are not set, the default is to export those columns as generic type GEOMETRY to be able to receive any spatial type. The AUTODETECT_SPATIAL_TYPE directive allows to force Ora2Pg to autodetect the real spatial type and dimension used in a spatial column otherwise a non- constrained "geometry" type is used. Enabling this feature will force Ora2Pg to scan a sample of 50000 column to look at the GTYPE used. You can increase or reduce the sample size by setting the value of AUTODETECT_SPATIAL_TYPE to the desired number of line to scan. The directive is enabled by default. For example, in the case of a column named shape and defined with Oracle type SDO_GEOMETRY, with AUTODETECT_SPATIAL_TYPE disabled it will be converted as: shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) and if the directive is enabled and the column just contains a single geometry type that use a single dimension: shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) with a two or three dimensional polygon. CONVERT_SRID This directive allows you to control the automatically conversion of Oracle SRID to standard EPSG. If enabled, Ora2Pg will use the Oracle function sdo_cs.map_oracle_srid_to_epsg() to convert all SRID. Enabled by default. If the SDO_SRID returned by Oracle is NULL, it will be replaced by the default value 8307 converted to its EPSG value: 4326 (see DEFAULT_SRID). If the value is upper than 1, all SRID will be forced to this value, in this case DEFAULT_SRID will not be used when Oracle returns a null value and the value will be forced to CONVERT_SRID. Note that it is also possible to set the EPSG value on Oracle side when sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to force the value: system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; DEFAULT_SRID Use this directive to override the default EPSG SRID to used: 4326. Can be overwritten by CONVERT_SRID, see above. GEOMETRY_EXTRACT_TYPE This directive can take three values: WKT (default), WKB and INTERNAL. When it is set to WKT, Ora2Pg will use SDO_UTIL.TO_WKTGEOMETRY() to extract the geometry data. When it is set to WKB, Ora2Pg will use the binary output using SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type are calls at Oracle side, they are slow and you can easily reach Out Of Memory when you have lot of rows. Also WKB is not able to export 3D geometry and some geometries like CURVEPOLYGON. In this case you may use the INTERNAL extraction type. It will use a Pure Perl library to convert the SDO_GEOMETRY data into a WKT representation, the translation is done on Ora2Pg side. This is a work in progress, please validate your exported data geometries before use. Default spatial object extraction type is INTERNAL. POSTGIS_SCHEMA Use this directive to add a specific schema to the search path to look for PostGis functions. ST_SRID_FUNCTION Oracle function to use to extract the srid from ST_Geometry meta information. Default: ST_SRID, for example it should be set to sde.st_srid for ArcSDE. ST_DIMENSION_FUNCTION Oracle function to use to extract the dimension from ST_Geometry meta information. Default: ST_DIMENSION, for example it should be set to sde.st_dimention for ArcSDE. ST_GEOMETRYTYPE_FUNCTION Oracle function to use to extract the geometry type from a ST_Geometry column Default: ST_GEOMETRYTYPE, for example it should be set to sde.st_geometrytype for ArcSDE. ST_ASBINARY_FUNCTION Oracle function to used to convert an ST_Geometry value into WKB format. Default: ST_ASBINARY, for example it should be set to sde.st_asbinary for ArcSDE. ST_ASTEXT_FUNCTION Oracle function to used to convert an ST_Geometry value into WKT format. Default: ST_ASTEXT, for example it should be set to sde.st_astext for ArcSDE. PostgreSQL Import By default conversion to PostgreSQL format is written to file 'output.sql'. The command: psql mydb < output.sql will import content of file output.sql into PostgreSQL mydb database. DATA_LIMIT When you are performing INSERT/COPY export Ora2Pg proceed by chunks of DATA_LIMIT tuples for speed improvement. Tuples are stored in memory before being written to disk, so if you want speed and have enough system resources you can grow this limit to an upper value for example: 100000 or 1000000. Before release 7.0 a value of 0 mean no limit so that all tuples are stored in memory before being flushed to disk. In 7.x branch this has been remove and chunk will be set to the default: 10000 BLOB_LIMIT When Ora2Pg detect a table with some BLOB it will automatically reduce the value of this directive by dividing it by 10 until his value is below 1000. You can control this value by setting BLOB_LIMIT. Exporting BLOB use lot of resources, setting it to a too high value can produce OOM. CLOB_AS_BLOB Apply same behavior on CLOB than BLOB with BLOB_LIMIT settings. This is especially useful if you have large CLOB data. Default: enabled OUTPUT The Ora2Pg output filename can be changed with this directive. Default value is output.sql. if you set the file name with extension .gz or .bz2 the output will be automatically compressed. This require that the Compress::Zlib Perl module is installed if the filename extension is .gz and that the bzip2 system command is installed for the .bz2 extension. OUTPUT_DIR Since release 7.0, you can define a base directory where the file will be written. The directory must exists. BZIP2 This directive allows you to specify the full path to the bzip2 program if it can not be found in the PATH environment variable. FILE_PER_CONSTRAINT Allow object constraints to be saved in a separate file during schema export. The file will be named CONSTRAINTS_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can use .gz xor .bz2 extension to enable compression. Default is to save all data in the OUTPUT file. This directive is usable only with TABLE export type. The constraints can be imported quickly into PostgreSQL using the LOAD export type to parallelize their creation over multiple (-j or JOBS) connections. FILE_PER_INDEX Allow indexes to be saved in a separate file during schema export. The file will be named INDEXES_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can use .gz xor .bz2 file extension to enable compression. Default is to save all data in the OUTPUT file. This directive is usable only with TABLE AND TABLESPACE export type. With the TABLESPACE export, it is used to write "ALTER INDEX ... TABLESPACE ..." into a separate file named TBSP_INDEXES_OUTPUT that can be loaded at end of the migration after the indexes creation to move the indexes. The indexes can be imported quickly into PostgreSQL using the LOAD export type to parallelize their creation over multiple (-j or JOBS) connections. FILE_PER_FKEYS Allow foreign key declaration to be saved in a separate file during schema export. By default foreign keys are exported into the main output file or in the CONSTRAINT_output.sql file. When enabled foreign keys will be exported into a file named FKEYS_output.sql FILE_PER_TABLE Allow data export to be saved in one file per table/view. The files will be named as tablename_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the OUTPUT directive to enable compression. Default 0 will save all data in one file, set it to 1 to enable this feature. This is usable only during INSERT or COPY export type. FILE_PER_FUNCTION Allow functions, procedures and triggers to be saved in one file per object. The files will be named as objectname_OUTPUT. Where OUTPUT is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the OUTPUT directive to enable compression. Default 0 will save all in one single file, set it to 1 to enable this feature. This is usable only during the corresponding export type, the package body export has a special behavior. When export type is PACKAGE and you've enabled this directive, Ora2Pg will create a directory per package, named with the lower case name of the package, and will create one file per function/procedure into that directory. If the configuration directive is not enabled, it will create one file per package as packagename_OUTPUT, where OUTPUT is the value of the corresponding directive. TRUNCATE_TABLE If this directive is set to 1, a TRUNCATE TABLE instruction will be add before loading data. This is usable only during INSERT or COPY export type. When activated, the instruction will be added only if there's no global DELETE clause or not one specific to the current table (see below). DELETE Support for include a DELETE FROM ... WHERE clause filter before importing data and perform a delete of some lines instead of truncating tables. Value is construct as follow: TABLE_NAME[DELETE_WHERE_CLAUSE], or if you have only one where clause for all tables just put the delete clause as single value. Both are possible too. Here are some examples: DELETE 1=1 # Apply to all tables and delete all tuples DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] The last applies two different delete where clause on tables TABLE_TEST and TABLE_INFO and a generic delete where clause on DATE_CREATE to all other tables. If TRUNCATE_TABLE is enabled it will be applied to all tables not covered by the DELETE definition. These DELETE clauses might be useful with regular "updates". STOP_ON_ERROR Set this parameter to 0 to not include the call to \set ON_ERROR_STOP ON in all SQL scripts generated by Ora2Pg. By default this order is always present so that the script will immediately abort when an error is encountered. COPY_FREEZE Enable this directive to use COPY FREEZE instead of a simple COPY to export data with rows already frozen. This is intended as a performance option for initial data loading. Rows will be frozen only if the table being loaded has been created or truncated in the current sub-transaction. This will only work with export to file and when -J or ORACLE_COPIES is not set or default to 1. It can be used with direct import into PostgreSQL under the same condition but -j or JOBS must also be unset or default to 1. CREATE_OR_REPLACE By default Ora2Pg uses CREATE OR REPLACE in functions and views DDL, if you need not to override existing functions or views disable this configuration directive, DDL will not include OR REPLACE. DROP_IF_EXISTS To add a DROP IF EXISTS before creating the object, enable this directive. Can be useful in an iterative work. Default is disabled. EXPORT_GTT PostgreSQL do not supports Global Temporary Table natively but you can use the pgtt extension to emulate this behavior. Enable this directive to export global temporary table. PGTT_NOSUPERUSER By default the pgtt extension is loaded using the superuser privilege. Enabled it if you run the SQL scripts generated using a non superuser user. It will use: LOAD '$libdir/plugins/pgtt'; instead of default: LOAD 'pgtt'; NO_HEADER Enabling this directive will prevent Ora2Pg to print his header into output files. Only the translated code will be written. PSQL_RELATIVE_PATH By default Ora2Pg use \i psql command to execute generated SQL files if you want to use a relative path following the script execution file enabling this option will use \ir. See psql help for more information. DATA_VALIDATION_ROWS Number of rows that must be retrieved on both side for data validation. Default it to compare the 10000 first rows. A value of 0 mean compare all rows. DATA_VALIDATION_ORDERING Order of rows between both sides are different once the data have been modified. In this case data must be ordered using a primary key or a unique index, that mean that a table without such object can not be compared. If the validation is done just after the data migration without any data modification the validation can be done on all tables without any ordering. DATA_VALIDATION_ERROR Stop validating data from a table after a certain amount of row mistmatch. Default is to stop after 10 rows validation errors. TRANSFORM_VALUE Use this directive to precise which transformation should be applied to a column when exporting data. Value must be a semicolon separated list of TABLE[COLUMN_NAME, ] For example to replace string 'Oracle' by 'PostgreSQL' in a varchar2 column use the following. TRANSFORM_VALUE ERROR_LOG_SAMPLE[DBMS_TYPE:regexp_replace("DBMS_TYPE",'Oracle','PostgreSQL')] or to replace all Oracle char(0) in a string by a space character: TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), ' ')] The expression will be applied in the SQL statemeent used to extract data from the source database. When using Ora2Pg export type INSERT or COPY to dump data to file and that FILE_PER_TABLE is enabled, you will be warned that Ora2Pg will not export data again if the file already exists. This is to prevent downloading twice table with huge amount of data. To force the download of data from these tables you have to remove the existing output file first. If you want to import data on the fly to the PostgreSQL database you have three configuration directives to set the PostgreSQL database connection. This is only possible with COPY or INSERT export type as for database schema there's no real interest to do that. PG_DSN Use this directive to set the PostgreSQL data source namespace using DBD::Pg Perl module as follow: dbi:Pg:dbname=pgdb;host=localhost;port=5432 will connect to database 'pgdb' on localhost at tcp port 5432. Note that this directive is only used for data export, other export need to be imported manually through the use og psql or any other PostgreSQL client. To use SSL encrypted connection you must add sslmode=require to the connection string like follow: dbi:Pg:dbname=pgdb;host=localhost;port=5432;sslmode=require PG_USER and PG_PWD These two directives are used to set the login user and password. If you do not supply a credential with PG_PWD and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If PG_USER is not set it will be asked interactively too. SYNCHRONOUS_COMMIT Specifies whether transaction commit will wait for WAL records to be written to disk before the command returns a "success" indication to the client. This is the equivalent to set synchronous_commit directive of postgresql.conf file. This is only used when you load data directly to PostgreSQL, the default is off to disable synchronous commit to gain speed at writing data. Some modified version of PostgreSQL, like greenplum, do not have this setting, so in this set this directive to 1, ora2pg will not try to change the setting. PG_INITIAL_COMMAND This directive can be used to send an initial command to PostgreSQL, just after the connection. For example to set some session parameters. This directive can be used multiple times. INSERT_ON_CONFLICT When enabled this instruct Ora2Pg to add an ON CONFLICT DO NOTHING clause to all INSERT statements generated for this type of data export. Column type control PG_NUMERIC_TYPE If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type NUMBER(p,s) is approximatively converted to real and float PostgreSQL data type. If you have monetary fields or don't want rounding issues with the extra decimals you should preserve the same numeric(p,s) PostgreSQL data type. Do that only if you need exactness because using numeric(p,s) is slower than using real or double. PG_INTEGER_TYPE If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type NUMBER(p) or NUMBER are converted to smallint, integer or bigint PostgreSQL data type following the value of the precision. If NUMBER without precision are set to DEFAULT_NUMERIC (see below). DEFAULT_NUMERIC NUMBER without precision are converted by default to bigint only if PG_INTEGER_TYPE is true. You can overwrite this value to any PG type, like integer or float. DATA_TYPE If you're experiencing any problem in data type schema conversion with this directive you can take full control of the correspondence between Oracle and PostgreSQL types to redefine data type translation used in Ora2pg. The syntax is a comma-separated list of "Oracle datatype:Postgresql datatype". Here are the default list used: DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,NVARCHAR:varchar,NCHAR:char,DATE:timestamp(0),LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW(16):uuid,RAW(32):uuid,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:integer,INTEGER:integer,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone The directive and the list definition must be a single line. Note that when a RAW(16) and RAW(32) columns is found or that the RAW column has "SYS_GUID()" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the "uuid-ossp" extension. If you want to replace a type with a precision and scale you need to escape the coma with a backslash. For example, if you want to replace all NUMBER(*,0) into bigint instead of numeric(38) add the following: DATA_TYPE NUMBER(*\,0):bigint You don't have to recopy all default type conversion but just the one you want to rewrite. There's a special case with BFILE when they are converted to type TEXT, they will just contains the full path to the external file. If you set the destination type to BYTEA, the default, Ora2Pg will export the content of the BFILE as bytea. The third case is when you set the destination type to EFILE, in this case, Ora2Pg will export it as an EFILE record: (DIRECTORY, FILENAME). Use the DIRECTORY export type to export the existing directories as well as privileges on those directories. There's no SQL function available to retrieve the path to the BFILE. Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); l_path VARCHAR2(4000); BEGIN dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); SELECT directory_path INTO l_path FROM all_directories WHERE directory_name = l_dir; l_dir := rtrim(l_path,'/'); RETURN l_dir || '/' || l_fname; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is TEXT. The function is dropped at the end of the export. This concern both, COPY and INSERT export type. There's no SQL function available to retrieve BFILE as an EFILE record, then Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); BEGIN dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); RETURN '(' || l_dir || ',' || l_fnamei || ')'; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is EFILE. The function is dropped at the end of the export. This concern both, COPY and INSERT export type. To set the destination type, use the DATA_TYPE configuration directive: DATA_TYPE BFILE:EFILE for example. The EFILE type is a user defined type created by the PostgreSQL extension external_file that can be found here: https://github.com/darold/external_file This is a port of the BFILE Oracle type to PostgreSQL. There's no SQL function available to retrieve the content of a BFILE. Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN BLOB AS filecontent BLOB := NULL; src_file BFILE := NULL; l_step PLS_INTEGER := 12000; l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); offset NUMBER := 1; BEGIN IF p_bfile IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); src_file := BFILENAME( l_dir, l_fname ); IF src_file IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); DBMS_LOB.CREATETEMPORARY(filecontent, true); DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); DBMS_LOB.FILECLOSE(src_file); RETURN filecontent; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is bytea (the default). The function is dropped at the end of the export. This concern both, COPY and INSERT export type. About the ROWID and UROWID, they are converted into OID by "logical" default but this will through an error at data import. There is no equivalent data type so you might want to use the DATA_TYPE directive to change the corresponding type in PostgreSQL. You should consider replacing this data type by a bigserial (autoincremented sequence), text or uuid data type. MODIFY_TYPE Sometimes you need to force the destination type, for example a column exported as timestamp by Ora2Pg can be forced into type date. Value is a comma-separated list of TABLE:COLUMN:TYPE structure. If you need to use comma or space inside type definition you will have to backslash them. MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) Type of table1.col3 will be replaced by a varchar and table1.col4 by a decimal with precision and scale. If the column's type is a user defined type Ora2Pg will autodetect the composite type and will export its data using ROW(). Some Oracle user defined types are just array of a native type, in this case you may want to transform this column in simple array of a PostgreSQL native type. To do so, just redefine the destination type as wanted and Ora2Pg will also transform the data as an array. For example, with the following definition in Oracle: CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); CREATE TABLE club (Name VARCHAR2(10), Address VARCHAR2(20), City VARCHAR2(20), Phone VARCHAR2(8), Members mem_type ); custom type "mem_type" is just a string array and can be translated into the following in PostgreSQL: CREATE TABLE club ( name varchar(10), address varchar(20), city varchar(20), phone varchar(8), members text[] ) ; To do so, just use the directive as follow: MODIFY_TYPE CLUB:MEMBERS:text[] Ora2Pg will take care to transform all data of this column in the correct format. Only arrays of characters and numerics types are supported. TO_NUMBER_CONVERSION By default Oracle call to function TO_NUMBER will be translated as a cast into numeric. For example, TO_NUMBER('10.1234') is converted into PostgreSQL call to_number('10.1234')::numeric. If you want you can cast the call to integer or bigint by changing the value of the configuration directive. If you need better control of the format, just set it as value, for example: TO_NUMBER_CONVERSION 99999999999999999999.9999999999 will convert the code above as: TO_NUMBER('10.1234', '99999999999999999999.9999999999') Any value of the directive that it is not numeric, integer or bigint will be taken as a mask format. If set to none, no conversion will be done. VARCHAR_TO_TEXT By default varchar2 without size constraint are tranlated into text. If you want to keep the varchar name, disable this directive. FORCE_IDENTITY_BIGINT Usually identity column must be bigint to correspond to an auto increment sequence so Ora2Pg always force it to be a bigint. If, for any reason you want Ora2Pg to respect the DATA_TYPE you have set for identity column then disable this directive. TO_CHAR_NOTIMEZONE If you want Ora2Pg to remove any timezone information into the format part of the TO_CHAR() function, enable this directive. Disabled by default. Taking export under control The following other configuration directives interact directly with the export process and give you fine granularity in database export control. SKIP For TABLE export you may not want to export all schema constraints, the SKIP configuration directive allows you to specify a space-separated list of constraints that should not be exported. Possible values are: - fkeys: turn off foreign key constraints - pkeys: turn off primary keys - ukeys: turn off unique column constraints - indexes: turn off all other index types - checks: turn off check constraints For example: SKIP indexes,checks will removed indexes and check constraints from export. PKEY_IN_CREATE Enable this directive if you want to add primary key definition inside the create table statement. If disabled (the default) primary key definition will be added with an alter table statement. Enable it if you are exporting to GreenPlum PostgreSQL database. KEEP_PKEY_NAMES By default names of the primary and unique key in the source Oracle database are ignored and key names are autogenerated in the target PostgreSQL database with the PostgreSQL internal default naming rules. If you want to preserve Oracle primary and unique key names set this option to 1. FKEY_ADD_UPDATE This directive allows you to add an ON UPDATE CASCADE option to a foreign key when a ON DELETE CASCADE is defined or always. Oracle do not support this feature, you have to use trigger to operate the ON UPDATE CASCADE. As PostgreSQL has this feature, you can choose how to add the foreign key option. There are three values to this directive: never, the default that mean that foreign keys will be declared exactly like in Oracle. The second value is delete, that mean that the ON UPDATE CASCADE option will be added only if the ON DELETE CASCADE is already defined on the foreign Keys. The last value, always, will force all foreign keys to be defined using the update option. FKEY_DEFERRABLE When exporting tables, Ora2Pg normally exports constraints as they are, if they are non-deferrable they are exported as non-deferrable. However, non-deferrable constraints will probably cause problems when attempting to import data to Pg. The FKEY_DEFERRABLE option set to 1 will cause all foreign key constraints to be exported as deferrable. DEFER_FKEY In addition to exporting data when the DEFER_FKEY option set to 1, it will add a command to defer all foreign key constraints during data export and the import will be done in a single transaction. This will work only if foreign keys have been exported as deferrable and you are not using direct import to PostgreSQL (PG_DSN is not defined). Constraints will then be checked at the end of the transaction. This directive can also be enabled if you want to force all foreign keys to be created as deferrable and initially deferred during schema export (TABLE export type). DROP_FKEY If deferring foreign keys is not possible due to the amount of data in a single transaction, you've not exported foreign keys as deferrable or you are using direct import to PostgreSQL, you can use the DROP_FKEY directive. It will drop all foreign keys before all data import and recreate them at the end of the import. DROP_INDEXES This directive allows you to gain lot of speed improvement during data import by removing all indexes that are not an automatic index (indexes of primary keys) and recreate them at the end of data import. Of course it is far better to not import indexes and constraints before having imported all data. DISABLE_TRIGGERS This directive is used to disable triggers on all tables in COPY or INSERT export modes. Available values are USER (disable user-defined triggers only) and ALL (includes RI system triggers). Default is 0: do not add SQL statements to disable trigger before data import. If you want to disable triggers during data migration, set the value to USER if your are connected as non superuser and ALL if you are connected as PostgreSQL superuser. A value of 1 is equal to USER. DISABLE_SEQUENCE If set to 1 it disables alter of sequences on all tables during COPY or INSERT export mode. This is used to prevent the update of sequence during data migration. Default is 0, alter sequences. NOESCAPE By default all data that are not of type date or time are escaped. If you experience any problem with that you can set it to 1 to disable character escaping during data export. This directive is only used during a COPY export. See STANDARD_CONFORMING_STRINGS for enabling/disabling escape with INSERT statements. STANDARD_CONFORMING_STRINGS This controls whether ordinary string literals ('...') treat backslashes literally, as specified in SQL standard. This was the default before Ora2Pg v8.5 so that all strings was escaped first, now this is currently on, causing Ora2Pg to use the escape string syntax (E'...') if this parameter is not set to 0. This is the exact behavior of the same option in PostgreSQL. This directive is only used during data export to build INSERT statements. See NOESCAPE for enabling/disabling escape in COPY statements. TRIM_TYPE If you want to convert CHAR(n) from Oracle into varchar(n) or text on PostgreSQL using directive DATA_TYPE, you might want to do some trimming on the data. By default Ora2Pg will auto-detect this conversion and remove any whitespace at both leading and trailing position. If you just want to remove the leadings character set the value to LEADING. If you just want to remove the trailing character, set the value to TRAILING. Default value is BOTH. TRIM_CHAR The default trimming character is space, use this directive if you need to change the character that will be removed. For example, set it to - if you have leading - in the char(n) field. To use space as trimming charger, comment this directive, this is the default value. PRESERVE_CASE If you want to preserve the case of Oracle object name set this directive to 1. By default Ora2Pg will convert all Oracle object names to lower case. I do not recommend to enable this unless you will always have to double-quote object names on all your SQL scripts. ORA_RESERVED_WORDS Allow escaping of column name using Oracle reserved words. Value is a list of comma-separated reserved word. Default: audit,comment,references. USE_RESERVED_WORDS Enable this directive if you have table or column names that are a reserved word for PostgreSQL. Ora2Pg will double quote the name of the object. GEN_USER_PWD Set this directive to 1 to replace default password by a random password for all extracted user during a GRANT export. PG_SUPPORTS_MVIEW Since PostgreSQL 9.3, materialized view are supported with the SQL syntax 'CREATE MATERIALIZED VIEW'. To force Ora2Pg to use the native PostgreSQL support you must enable this configuration - enable by default. If you want to use the old style with table and a set of function, you should disable it. PG_SUPPORTS_IFEXISTS PostgreSQL version below 9.x do not support IF EXISTS in DDL statements. Disabling the directive with value 0 will prevent Ora2Pg to add those keywords in all generated statements. Default value is 1, enabled. PG_VERSION Set the PostgreSQL major version number of the target database. Ex: 9.6 or 13 Default is current major version at time of a new release. This replace the old and deprecadted PG_SUPPORTS_* configuration directives described bellow. PG_SUPPORTS_ROLE (Deprecated) This option is deprecated since Ora2Pg release v7.3. By default Oracle roles are translated into PostgreSQL groups. If you have PostgreSQL 8.1 or more consider the use of ROLES and set this directive to 1 to export roles. PG_SUPPORTS_INOUT (Deprecated) This option is deprecated since Ora2Pg release v7.3. If set to 0, all IN, OUT or INOUT parameters will not be used into the generated PostgreSQL function declarations (disable it for PostgreSQL database version lower than 8.1), This is now enable by default. PG_SUPPORTS_DEFAULT This directive enable or disable the use of default parameter value in function export. Until PostgreSQL 8.4 such a default value was not supported, this feature is now enable by default. PG_SUPPORTS_WHEN (Deprecated) Add support to WHEN clause on triggers as PostgreSQL v9.0 now support it. This directive is enabled by default, set it to 0 disable this feature. PG_SUPPORTS_INSTEADOF (Deprecated) Add support to INSTEAD OF usage on triggers (used with PG >= 9.1), if this directive is disabled the INSTEAD OF triggers will be rewritten as Pg rules. PG_SUPPORTS_CHECKOPTION When enabled, export views with CHECK OPTION. Disable it if you have PostgreSQL version prior to 9.4. Default: 1, enabled. PG_SUPPORTS_IFEXISTS If disabled, do not export object with IF EXISTS statements. Enabled by default. PG_SUPPORTS_PARTITION PostgreSQL version prior to 10.0 do not have native partitioning. Enable this directive if you want to use declarative partitioning. Enable by default. PG_SUPPORTS_SUBSTR Some versions of PostgreSQL like Redshift doesn't support substr() and it need to be replaced by a call to substring(). In this case, disable it. PG_SUPPORTS_NAMED_OPERATOR Disable this directive if you are using PG < 9.5, PL/SQL operator used in named parameter => will be replaced by PostgreSQL proprietary operator := Enable by default. PG_SUPPORTS_IDENTITY Enable this directive if you have PostgreSQL >= 10 to use IDENTITY columns instead of serial or bigserial data type. If PG_SUPPORTS_IDENTITY is disabled and there is IDENTITY column in the Oracle table, they are exported as serial or bigserial columns. When it is enabled they are exported as IDENTITY columns like: CREATE TABLE identity_test_tab ( id bigint GENERATED ALWAYS AS IDENTITY, description varchar(30) ) ; If there is non default sequence options set in Oracle, they will be appended after the IDENTITY keyword. Additionally in both cases, Ora2Pg will create a file AUTOINCREMENT_output.sql with a embedded function to update the associated sequences with the restart value set to "SELECT max(colname)+1 FROM tablename". Of course this file must be imported after data import otherwise sequence will be kept to start value. Enabled by default. PG_SUPPORTS_PROCEDURE PostgreSQL v11 adds support of PROCEDURE, enable it if you use such version. BITMAP_AS_GIN Use btree_gin extension to create bitmap like index with pg >= 9.4 You will need to create the extension by yourself: create extension btree_gin; Default is to create GIN index, when disabled, a btree index will be created PG_BACKGROUND Use pg_background extension to create an autonomous transaction instead of using a dblink wrapper. With pg >= 9.5 only. Default is to use dblink. See https://github.com/vibhorkum/pg_background about this extension. DBLINK_CONN By default if you have an autonomous transaction translated using dblink extension instead of pg_background the connection is defined using the values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override the connection string use this directive as follow to set the connection in the autonomous transaction wrapper function. For example: DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass LONGREADLEN Use this directive to set the database handle's 'LongReadLen' attribute to a value that will be the larger than the expected size of the LOBs. The default is 1MB witch may not be enough to extract BLOBs or CLOBs. If the size of the LOB exceeds the 'LongReadLen' DBD::Oracle will return a 'ORA-24345: A Truncation' error. Default: 1023*1024 bytes. Take a look at this page to learn more: http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Inter face_for_Persistent_LOBs Important note: If you increase the value of this directive take care that DATA_LIMIT will probably needs to be reduced. Even if you only have a 1MB blob, trying to read 10000 of them (the default DATA_LIMIT) all at once will require 10GB of memory. You may extract data from those table separately and set a DATA_LIMIT to 500 or lower, otherwise you may experience some out of memory. LONGTRUNKOK If you want to bypass the 'ORA-24345: A Truncation' error, set this directive to 1, it will truncate the data extracted to the LongReadLen value. Disable by default so that you will be warned if your LongReadLen value is not high enough. USE_LOB_LOCATOR Disable this if you want to load full content of BLOB and CLOB and not use LOB locators. In this case you will have to set LONGREADLEN to the right value. Note that this will not improve speed of BLOB export as most of the time is always consumed by the bytea escaping and in this case export is done line by line and not by chunk of DATA_LIMIT rows. For more information on how it works, see http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Da ta_Interface_for_LOB_Locators Default is enabled, it use LOB locators. LOB_CHUNK_SIZE Oracle recommends reading from and writing to a LOB in batches using a multiple of the LOB chunk size. This chunk size defaults to 8k (8192). Recent tests shown that the best performances can be reach with higher value like 512K or 4Mb. A quick benchmark with 30120 rows with different size of BLOB (200x5Mb, 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, LONGREADLEN=170Mb and a total table size of 20GB gives: no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE set to 4Mb. Depending of the size of most BLOB you may want to adjust the value here. For example if you have a majority of small lobs bellow 8K, using 8192 is better to not waste space. Default value for LOB_CHUNK_SIZE is 512000. XML_PRETTY Force the use getStringVal() instead of getClobVal() for XML data export. Default is 1, enabled for backward compatibility. Set it to 0 to use extract method a la CLOB. Note that XML value extracted with getStringVal() must not exceed VARCHAR2 size limit (4000) otherwise it will return an error. ENABLE_MICROSECOND Set it to O if you want to disable export of millisecond from Oracle timestamp columns. By default milliseconds are exported with the use of following format: 'YYYY-MM-DD HH24:MI:SS.FF' Disabling will force the use of the following Oracle format: to_char(..., 'YYYY-MM-DD HH24:MI:SS') By default milliseconds are exported. DISABLE_COMMENT Set this to 1 if you don't want to export comment associated to tables and columns definition. Default is enabled. Control MySQL export behavior MYSQL_PIPES_AS_CONCAT Enable this if double pipe and double ampersand (|| and &&) should not be taken as equivalent to OR and AND. It depend of the variable @sql_mode, Use it only if Ora2Pg fail on auto detecting this behavior. MYSQL_INTERNAL_EXTRACT_FORMAT Enable this directive if you want EXTRACT() replacement to use the internal format returned as an integer, for example DD HH24:MM:SS will be replaced with format; DDHH24MMSS::bigint, this depend of your apps usage. Control SQL Server export behavior DROP_ROWVERSION PostgreSQL has no equivalent to rowversion datatype and feature, if you want to remove these useless columns, enable this directive. Columns of datatype 'rowversion' or 'timestamp' will not be exported. CASE_INSENSITIVE_SEARCH Emulate the same behavior of MSSQL with case insensitive search. If the value is citext it will use the citext data type instead of char/varchar/text in tables DDL (Ora2Pg will add a CHECK constraint for columns with a precision). Instead of citext you can also set a collation name that will be used in the columns definitions. To disable case insensitive search set it to: none. SELECT_TOP Append a TOP N clause to the SELECT command used to extract the data from SQL Server. This is the equivalent to a WHERE ROWNUM < 1000 clause for Oracle. Special options to handle character encoding NLS_LANG and NLS_NCHAR By default Ora2Pg will set NLS_LANG to AMERICAN_AMERICA.AL32UTF8 and NLS_NCHAR to AL32UTF8. It is not recommended to change those settings but in some case it could be useful. Using your own settings with those configuration directive will change the client encoding at Oracle side by setting the environment variables $ENV{NLS_LANG} and $ENV{NLS_NCHAR}. BINMODE By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through a call to the Perl pragma: use open ':utf8'; You can override this encoding by using the BINMODE directive, for example you can set it to :locale to use your locale or iso-8859-7, it will respectively use use open ':locale'; use open ':encoding(iso-8859-7)'; If you have change the NLS_LANG in non UTF8 encoding, you might want to set this directive. See http://perldoc.perl.org/5.14.2/open.html for more information. Most of the time, leave this directive commented. CLIENT_ENCODING By default PostgreSQL client encoding is automatically set to UTF8 to avoid encoding issue. If you have changed the value of NLS_LANG you might have to change the encoding of the PostgreSQL client. You can take a look at the PostgreSQL supported character sets here: http://www.postgresql.org/docs/9.0/static/multibyte.html FORCE_PLSQL_ENCODING To force utf8 encoding of the PL/SQL code exported, enable this directive. Could be helpful in some rare condition. PLSQL to PLPGSQL conversion Automatic code conversion from Oracle PLSQL to PostgreSQL PLPGSQL is a work in progress in Ora2Pg and surely you will always have manual work. The Perl code used for automatic conversion is all stored in a specific Perl Module named Ora2Pg/PLSQL.pm feel free to modify/add you own code and send me patches. The main work in on function, procedure, package and package body headers and parameters rewrite. PLSQL_PGSQL Enable/disable PLSQL to PLPGSQL conversion. Enabled by default. NULL_EQUAL_EMPTY Ora2Pg can replace all conditions with a test on NULL by a call to the coalesce() function to mimic the Oracle behavior where empty string are considered equal to NULL. (field1 IS NULL) is replaced by (coalesce(field1::text, '') = '') (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> '') You might want this replacement to be sure that your application will have the same behavior but if you have control on you application a better way is to change it to transform empty string into NULL because PostgreSQL makes the difference. EMPTY_LOB_NULL Force empty_clob() and empty_blob() to be exported as NULL instead as empty string for the first one and '\x' for the second. If NULL is allowed in your column this might improve data export speed if you have lot of empty lob. Default is to preserve the exact data from Oracle. PACKAGE_AS_SCHEMA If you don't want to export package as schema but as simple functions you might also want to replace all call to package_name.function_name. If you disable the PACKAGE_AS_SCHEMA directive then Ora2Pg will replace all call to package_name.function_name() by package_name_function_name(). Default is to use a schema to emulate package. The replacement will be done in all kind of DDL or code that is parsed by the PLSQL to PLPGSQL converter. PLSQL_PGSQL must be enabled or -p used in command line. REWRITE_OUTER_JOIN Enable this directive if the rewrite of Oracle native syntax (+) of OUTER JOIN is broken. This will force Ora2Pg to not rewrite such code, default is to try to rewrite simple form of right outer join for the moment. UUID_FUNCTION By default Ora2Pg will convert call to SYS_GUID() Oracle function with a call to uuid_generate_v4 from uuid-ossp extension. You can redefined it to use the gen_random_uuid function from pgcrypto extension by changing the function name. Default to uuid_generate_v4. Note that when a RAW(16) and RAW(32) columns is found or that the RAW column has "SYS_GUID()" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the "uuid-ossp" extension. FUNCTION_STABLE By default Oracle functions are marked as STABLE as they can not modify data unless when used in PL/SQL with variable assignment or as conditional expression. You can force Ora2Pg to create these function as VOLATILE by disabling this configuration directive. COMMENT_COMMIT_ROLLBACK By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to force the user to review the logic of the function. Once it is fixed in Oracle source code or you want to comment this calls enable the following directive. COMMENT_SAVEPOINT It is common to see SAVEPOINT call inside PL/SQL procedure together with a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is enabled you may want to also comment SAVEPOINT calls, in this case enable it. STRING_CONSTANT_REGEXP Ora2Pg replace all string constant during the pl/sql to plpgsql translation, string constant are all text include between single quote. If you have some string placeholder used in dynamic call to queries you can set a list of regexp to be temporary replaced to not break the parser. For example: STRING_CONSTANT_REGEXP The list of regexp must use the semi colon as separator. ALTERNATIVE_QUOTING_REGEXP To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals set the regexp with the text capture to use to extract the text part. For example with a variable declared as c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; the regexp to use must be: ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' ora2pg will use the $$ delimiter, with the example the result will be: c_sample varchar(100) := $$This doesn't work.$$; The value of this configuration directive can be a list of regexp separated by a semi colon. The capture part (between parenthesis) is mandatory in each regexp if you want to restore the string constant. USE_ORAFCE If you want to use functions defined in the Orafce library and prevent Ora2Pg to translate call to these functions, enable this directive. The Orafce library can be found here: https://github.com/orafce/orafce By default Ora2pg rewrite add_month(), add_year(), date_trunc() and to_char() functions, but you may prefer to use the orafce version of these function that do not need any code transformation. AUTONOMOUS_TRANSACTION Enable translation of autonomous transactions into a wrapper function using dblink or pg_background extension. If you don't want to use this translation and just want the function to be exported as a normal one without the pragma call, disable this directive. Materialized view Materialized views are exported as snapshot "Snapshot Materialized Views" as PostgreSQL only supports full refresh. If you want to import the materialized views in PostgreSQL prior to 9.3 you have to set configuration directive PG_SUPPORTS_MVIEW to 0. In this case Ora2Pg will export all materialized views as explain in this document: http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. When exporting materialized view Ora2Pg will first add the SQL code to create the "materialized_views" table: CREATE TABLE materialized_views ( mview_name text NOT NULL PRIMARY KEY, view_name text NOT NULL, iname text, last_refresh TIMESTAMP WITH TIME ZONE ); all materialized views will have an entry in this table. It then adds the plpgsql code to create tree functions: create_materialized_view(text, text, text) used to create a materialized view drop_materialized_view(text) used to delete a materialized view refresh_full_materialized_view(text) used to refresh a view then it adds the SQL code to create the view and the materialized view: CREATE VIEW mviewname_mview AS SELECT ... FROM ...; SELECT create_materialized_view('mviewname','mviewname_mview', change with the name of the column to used for the index); The first argument is the name of the materialized view, the second the name of the view on which the materialized view is based and the third is the column name on which the index should be build (aka most of the time the primary key). This column is not automatically deduced so you need to replace its name. As said above Ora2Pg only supports snapshot materialized views so the table will be entirely refreshed by issuing first a truncate of the table and then by load again all data from the view: refresh_full_materialized_view('mviewname'); To drop the materialized view you just have to call the drop_materialized_view() function with the name of the materialized view as parameter. Other configuration directives DEBUG Set it to 1 will enable verbose output. IMPORT You can define common Ora2Pg configuration directives into a single file that can be imported into other configuration files with the IMPORT configuration directive as follow: IMPORT commonfile.conf will import all configuration directives defined into commonfile.conf into the current configuration file. Exporting views as PostgreSQL tables You can export any Oracle view as a PostgreSQL table simply by setting TYPE configuration option to TABLE to have the corresponding create table statement. Or use type COPY or INSERT to export the corresponding data. To allow that you have to specify your views in the VIEW_AS_TABLE configuration option. Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) into a PG create table form, then it will extract the data (if TYPE=COPY or INSERT) following the view schema. For example, with the following view: CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS SELECT category_id, COUNT(*) as product_count, MIN(list_price) as low_price, MAX(list_price) as high_price FROM product_information GROUP BY category_id; Setting VIEW_AS_TABLE to product_prices and using export type TABLE, will force Ora2Pg to detect columns returned types and to generate a create table statement: CREATE TABLE product_prices ( category_id bigint, product_count integer, low_price numeric, high_price numeric ); Data will be loaded following the COPY or INSERT export type and the view declaration. You can use the ALLOW and EXCLUDE directive in addition to filter other objects to export. Export as Kettle transformation XML files The KETTLE export type is useful if you want to use Penthalo Data Integrator (Kettle) to import data to PostgreSQL. With this type of export Ora2Pg will generate one XML Kettle transformation files (.ktr) per table and add a line to manually execute the transformation in the output.sql file. For example: ora2pg -c ora2pg.conf -t KETTLE -j 12 -a MYTABLE -o load_mydata.sh will generate one file called 'HR.MYTABLE.ktr' and add a line to the output file (load_mydata.sh): #!/bin/sh KETTLE_TEMPLATE_PATH='.' JAVAMAXMEM=4096 ./pan.sh -file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr -level Detailed The -j 12 option will create a template with 12 processes to insert data into PostgreSQL. It is also possible to specify the number of parallel queries used to extract data from the Oracle with the -J command line option as follow: ora2pg -c ora2pg.conf -t KETTLE -J 4 -j 12 -a EMPLOYEES -o load_mydata.sh This is only possible if there is a unique key defined on a numeric column or that you have defined the technical key to used to split the query between cores in the DEFINED_PKEY configuration directive. For example: DEFINED_PK EMPLOYEES:employee_id will force the number of Oracle connection copies to 4 and defined the SQL query as follow in the Kettle XML transformation file: SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} The KETTLE export type requires that the Oracle and PostgreSQL DSN are defined. You can also activate the TRUNCATE_TABLE directive to force a truncation of the table before data import. The KETTLE export type is an original work of Marc Cousin. Migration cost assessment Estimating the cost of a migration process from Oracle to PostgreSQL is not easy. To obtain a good assessment of this migration cost, Ora2Pg will inspect all database objects, all functions and stored procedures to detect if there's still some objects and PL/SQL code that can not be automatically converted by Ora2Pg. Ora2Pg has a content analysis mode that inspect the Oracle database to generate a text report on what the Oracle database contains and what can not be exported. To activate the "analysis and report" mode, you have to use the export de type SHOW_REPORT like in the following command: ora2pg -t SHOW_REPORT Here is a sample report obtained with this command: -------------------------------------- Ora2Pg: Oracle Database Content Report -------------------------------------- Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 Schema HR Size 880.00 MB -------------------------------------- Object Number Invalid Comments -------------------------------------- CLUSTER 2 0 Clusters are not supported and will not be exported. FUNCTION 40 0 Total size of function code: 81992. INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL. 1 bitmap index(es). 230 b-tree index(es). 1 reversed b-tree index(es) Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed. PACKAGE BODY 2 1 Total size of package code: 20700. PROCEDURE 7 0 Total size of procedure code: 19198. SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files. 2 binary columns. 4 unknown types. TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. TRIGGER 30 0 Total size of trigger code: 21677. TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are converted as table, type inheritance is not supported. TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. Once the database can be analysed, Ora2Pg, by his ability to convert SQL and PL/SQL code from Oracle syntax to PostgreSQL, can go further by estimating the code difficulties and estimate the time necessary to operate a full database migration. To estimate the migration cost in person-days, Ora2Pg allow you to use a configuration directive called ESTIMATE_COST that you can also enabled at command line: --estimate_cost This feature can only be used with the SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE and QUERY export type. ora2pg -t SHOW_REPORT --estimate_cost The generated report is same as above but with a new 'Estimated cost' column as follow: -------------------------------------- Ora2Pg: Oracle Database Content Report -------------------------------------- Version Oracle Database 10g Express Edition Release 10.2.0.1.0 Schema HR Size 890.00 MB -------------------------------------- Object Number Invalid Estimated cost Comments -------------------------------------- DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions using oracle_fdw. FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL. 11 b-tree index(es). Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. JOB 0 0 0 Job are not exported. You may set external cron job with them. MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed. PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround is to use views or set the PostgreSQL search_path in your session to access object outside the current schema. user1.emp_details_view_v is an alias to hr.emp_details_view. user1.emp_table is an alias to hr.employees@other_server. user1.offices is an alias to hr.locations. TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files. 2 binary columns. 4 unknown types. TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are converted as table, type inheritance is not supported. TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. -------------------------------------- Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). The last line shows the total estimated migration cost in person-days following the number of migration units estimated for each object. This migration unit represent around five minutes for a PostgreSQL expert. If this is your first migration you can get it higher with the configuration directive COST_UNIT_VALUE or the --cost_unit_value command line option: ora2pg -t SHOW_REPORT --estimate_cost --cost_unit_value 10 Ora2Pg is also able to give you a migration difficulty level assessment, here a sample: Migration level: B-5 Migration levels: A - Migration that might be run automatically B - Migration with code rewrite and a person-days cost up to 5 days C - Migration with code rewrite and a person-days cost above 5 days Technical levels: 1 = trivial: no stored functions and no triggers 2 = easy: no stored functions but with triggers, no manual rewriting 3 = simple: stored functions and/or triggers, no manual rewriting 4 = manual: no stored functions but with triggers or views with code rewriting 5 = difficult: stored functions and/or triggers with code rewriting This assessment consist in a letter A or B to specify if the migration needs manual rewriting or not. And a number from 1 up to 5 to give you a technical difficulty level. You have an additional option --human_days_limit to specify the number of person-days limit where the migration level should be set to C to indicate that it need a huge amount of work and a full project management with migration support. Default is 10 person-days. You can use the configuration directive HUMAN_DAYS_LIMIT to change this default value permanently. This feature has been developed to help you or your boss to decide which database to migrate first and the team that must be mobilized to operate the migration. Global Oracle and MySQL migration assessment Ora2Pg come with a script ora2pg_scanner that can be used when you have a huge number of instances and schema to scan for migration assessment. Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] -b | --binpath DIR: full path to directory where the ora2pg binary stays. Might be useful only on Windows OS. -c | --config FILE: set custom configuration file to use otherwise ora2pg will use the default: /etc/ora2pg/ora2pg.conf. -l | --list FILE : CSV file containing a list of databases to scan with all required information. The first line of the file can contain the following header that describes the format that must be used: "type","schema/database","dsn","user","password" -o | --outdir DIR : (optional) by default all reports will be dumped to a directory named 'output', it will be created automatically. If you want to change the name of this directory, set the name at second argument. -t | --test : just try all connections by retrieving the required schema or database name. Useful to validate your CSV list file. -u | --unit MIN : redefine globally the migration cost unit value in minutes. Default is taken from the ora2pg.conf (default 5 minutes). Here is a full example of a CSV databases list file: "type","schema/database","dsn","user","password" "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","HR","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","system","manager" The CSV field separator must be a comma. Note that if you want to scan all schemas from an Oracle instance you just have to leave the schema field empty, Ora2Pg will automatically detect all available schemas and generate a report for each one. Of course you need to use a connection user with enough privileges to be able to scan all schemas. For example: "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","usrname","passwd" will generate a report for all schema in the XE instance. Note that in this case the SCHEMA directive in ora2pg.conf must not be set. It will generate a CSV file with the assessment result, one line per schema or database and a detailed HTML report for each database scanned. Hint: Use the -t | --test option before to test all your connections in your CSV file. For Windows users you must use the -b command line option to set the directory where ora2pg_scanner stays otherwise the ora2pg command calls will fail. In the migration assessment details about functions Ora2Pg always include per default 2 migration units for TEST and 1 unit for SIZE per 1000 characters in the code. This mean that by default it will add 15 minutes in the migration assessment per function. Obviously if you have unitary tests or very simple functions this will not represent the real migration time. Migration assessment method Migration unit scores given to each type of Oracle database object are defined in the Perl library lib/Ora2Pg/PLSQL.pm in the %OBJECT_SCORE variable definition. The number of PL/SQL lines associated to a migration unit is also defined in this file in the $SIZE_SCORE variable value. The number of migration units associated to each PL/SQL code difficulties can be found in the same Perl library lib/Ora2Pg/PLSQL.pm in the hash %UNCOVERED_SCORE initialization. This assessment method is a work in progress so I'm expecting feedbacks on migration experiences to polish the scores/units attributed in those variables. Improving indexes and constraints creation speed Using the LOAD export type and a file containing SQL orders to perform, it is possible to dispatch those orders over multiple PostgreSQL connections. To be able to use this feature, the PG_DSN, PG_USER and PG_PWD must be set. Then: ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 will dispatch indexes creation over 4 simultaneous PostgreSQL connections. This will considerably accelerate this part of the migration process with huge data size. Exporting LONG RAW If you still have columns defined as LONG RAW, Ora2Pg will not be able to export these kind of data. The OCI library fail to export them and always return the same first record. To be able to export the data you need to transform the field as BLOB by creating a temporary table before migrating data. For example, the Oracle table: SQL> DESC TEST_LONGRAW Name NULL ? Type -------------------- -------- ---------------------------- ID NUMBER C1 LONG RAW need to be "translated" into a table using BLOB as follow: CREATE TABLE test_blob (id NUMBER, c1 BLOB); And then copy the data with the following INSERT query: INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; Then you just have to exclude the original table from the export (see EXCLUDE directive) and to renamed the new temporary table on the fly using the REPLACE_TABLES configuration directive. Global variables Oracle allow the use of global variables defined in packages. Ora2Pg will export these variables for PostgreSQL as user defined custom variables available in a session. Oracle variables assignment are exported as call to: PERFORM set_config('pkgname.varname', value, false); Use of these variables in the code is replaced by: current_setting('pkgname.varname')::global_variables_type; where global_variables_type is the type of the variable extracted from the package definition. If the variable is a constant or have a default value assigned at declaration, Ora2Pg will create a file global_variables.conf with the definition to include in the postgresql.conf file so that their values will already be set at database connection. Note that the value can always modified by the user so you can not have exactly a constant. Hints Converting your queries with Oracle style outer join (+) syntax to ANSI standard SQL at the Oracle side can save you lot of time for the migration. You can use TOAD Query Builder can re-write these using the proper ANSI syntax, see: http://www.toadworld.com/products/toad-for-oracle/f/10/t/9518.aspx There's also an alternative with SQL Developer Data Modeler, see http://www.thatjeffsmith.com/archive/2012/01/sql-developer-data-modeler- quick-tip-use-oracle-join-syntax-or-ansi/ Toad is also able to rewrite the native Oracle DECODE() syntax into ANSI standard SQL CASE statement. You can find some slide about this in a presentation given at PgConf.RU: http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf Test the migration The type of action called TEST allow you to check that all objects from Oracle database have been created under PostgreSQL. Of course PG_DSN must be set to be able to check PostgreSQL side. Note that this feature respect the schema name limitation if EXPORT_SCHEMA and SCHEMA or PG_SCHEMA are defined. If only EXPORT_SCHEMA is set all schemes from Oracle and PostgreSQL are scanned. You can filter to a single schema using SCHEMA and/or PG_SCHEMA but you can not filter on a list of schema. To test a list of schema you will have to repeat the calls to Ora2Pg by specifying a single schema each time. For example command: ora2pg -t TEST -c config/ora2pg.conf > migration_diff.txt Will create a file containing the report of all object and row count on both side, Oracle and PostgreSQL, with an error section giving you the detail of the differences for each kind of object. Here is a sample result: [TEST INDEXES COUNT] ORACLEDB:DEPARTMENTS:2 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:6 POSTGRES:employees:6 [ERRORS INDEXES COUNT] Table departments doesn't have the same number of indexes in Oracle (2) and in PostgreSQL (1). [TEST UNIQUE CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS UNIQUE CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of unique constraints. [TEST PRIMARY KEYS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS PRIMARY KEYS COUNT] OK, Oracle and PostgreSQL have the same number of primary keys. [TEST CHECK CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS CHECK CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of check constraints. [TEST NOT NULL CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS NOT NULL CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of not null constraints. [TEST COLUMN DEFAULT VALUE COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS COLUMN DEFAULT VALUE COUNT] OK, Oracle and PostgreSQL have the same number of column default value. [TEST IDENTITY COLUMN COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:0 POSTGRES:employees:0 [ERRORS IDENTITY COLUMN COUNT] OK, Oracle and PostgreSQL have the same number of identity column. [TEST FOREIGN KEYS COUNT] ORACLEDB:DEPARTMENTS:0 POSTGRES:departments:0 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS FOREIGN KEYS COUNT] OK, Oracle and PostgreSQL have the same number of foreign keys. [TEST TABLE COUNT] ORACLEDB:TABLE:2 POSTGRES:TABLE:2 [ERRORS TABLE COUNT] OK, Oracle and PostgreSQL have the same number of TABLE. [TEST TABLE TRIGGERS COUNT] ORACLEDB:DEPARTMENTS:0 POSTGRES:departments:0 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS TABLE TRIGGERS COUNT] OK, Oracle and PostgreSQL have the same number of table triggers. [TEST TRIGGER COUNT] ORACLEDB:TRIGGER:2 POSTGRES:TRIGGER:2 [ERRORS TRIGGER COUNT] OK, Oracle and PostgreSQL have the same number of TRIGGER. [TEST VIEW COUNT] ORACLEDB:VIEW:1 POSTGRES:VIEW:1 [ERRORS VIEW COUNT] OK, Oracle and PostgreSQL have the same number of VIEW. [TEST MVIEW COUNT] ORACLEDB:MVIEW:0 POSTGRES:MVIEW:0 [ERRORS MVIEW COUNT] OK, Oracle and PostgreSQL have the same number of MVIEW. [TEST SEQUENCE COUNT] ORACLEDB:SEQUENCE:1 POSTGRES:SEQUENCE:0 [ERRORS SEQUENCE COUNT] SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). [TEST TYPE COUNT] ORACLEDB:TYPE:1 POSTGRES:TYPE:0 [ERRORS TYPE COUNT] TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). [TEST FDW COUNT] ORACLEDB:FDW:0 POSTGRES:FDW:0 [ERRORS FDW COUNT] OK, Oracle and PostgreSQL have the same number of FDW. [TEST FUNCTION COUNT] ORACLEDB:FUNCTION:3 POSTGRES:FUNCTION:3 [ERRORS FUNCTION COUNT] OK, Oracle and PostgreSQL have the same number of functions. [TEST SEQUENCE VALUES] ORACLEDB:EMPLOYEES_NUM_SEQ:1285 POSTGRES:employees_num_seq:1285 [ERRORS SEQUENCE VALUES COUNT] OK, Oracle and PostgreSQL have the same values for sequences [TEST ROWS COUNT] ORACLEDB:DEPARTMENTS:27 POSTGRES:departments:27 ORACLEDB:EMPLOYEES:854 POSTGRES:employees:854 [ERRORS ROWS COUNT] OK, Oracle and PostgreSQL have the same number of rows. Data validation Data validation consists in comparing data retrieved from a foreign table pointing to the source Oracle table and a local PostgreSQL table resulting from the data export. To run data validation you can use a direct connection like any other Ora2Pg action but you can also use the oracle_fdw, mysql_fdw ior tds_fdw extension provided that FDW_SERVER and PG_DSN configuration directives are set. By default Ora2Pg will extract the 10000 first rows from both side, you can change this value using directive DATA_VALIDATION_ROWS. When it is set to zero all rows of the tables will be compared. Data validation requires that the table has a primary key or unique index and that the key columns is not a LOB. Rows will be sorted using this unique key. Due to differences in sort behavior between Oracle and PostgreSQL, if the collation of unique key columns in PostgreSQL is not 'C', the sort order can be different compared to Oracle. In this case the data validation will fail. Data validation must be done before any data is modified. Ora2Pg will stop comparing two tables after DATA_VALIDATION_ROWS is reached or that 10 errors has been encountered, result is dumped in a file named "data_validation.log" written in the current directory by default. The number of error before stopping the diff between rows can be controlled using the configuration directive DATA_VALIDATION_ERROR. All rows in errors are printed to the output file for your analyze. It is possible to parallelize data validation by using -P option or the corresponding configuration directive PARALLEL_TABLES in ora2pg.conf. Use of System Change Number (SCN) Ora2Pg is able to export data as of a specific SCN. You can set it at command line using the -S or --scn option. You can give a specific SCN or if you want to use the current SCN at first connection time set the value to 'current'. In this last case the connection user has the "SELECT ANY DICTIONARY" or the "SELECT_CATALOG_ROLE" role, the current SCN is looked at the v$database view. Example of use: ora2pg -c ora2pg.conf -t COPY --scn 16605281 This adds the following clause to the query used to retrieve data for example: AS OF SCN 16605281 You can also use th --scn option to use the Oracle flashback capabality by specifying a timestamp expression instead of a SCN. For example: ora2pg -c ora2pg.conf -t COPY --scn "TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS')" This will add the following clause to the query used to retrieve data: AS OF TIMESTAMP TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS') or for example to only retrive yesterday's data: ora2pg -c ora2pg.conf -t COPY --scn "SYSDATE - 1" Change Data Capture (CDC) Ora2Pg do not have such feature which allow to import data and to only apply changes after the first import. But you can use the --cdc_ready option to export data with registration of the SCN at the time of the table export. All SCN per tables are written to a file named TABLES_SCN.log by default, it can be changed using -C | --cdc_file option. These SCN registered per table during COPY or INSERT export can be used with a CDC tool. The format of the file is tablename:SCN per line. Importing BLOB as large objects By default Ora2Pg imports Oracle BLOB as bytea, the destination column is created using the bytea data type. If you want to use large object instead of bytea, just add the --blob_to_lo option to the ora2pg command. It will create the destination column as data type Oid and will save the BLOB as a large object using the lo_from_bytea() function. The Oid returned by the call to lo_from_bytea() is inserted in the destination column instead of a bytea. Because of the use of the function this option can only be used with actions SHOW_COLUMN, TABLE and INSERT. Action COPY is not allowed. If you want to use COPY or have huge size BLOB ( > 1GB) than can not be imported using lo_from_bytea() you can add option --lo_import to the ora2pg command. This will allow to import data in two passes. 1) Export data using COPY or INSERT will set the Oid destination column for BLOB to value 0 and save the BLOB value into a dedicated file. It will also create a Shell script to import the BLOB files into the database using psql command \lo_import and to update the table Oid column to the returned large object Oid. The script is named lo_import-TABLENAME.sh 2) Execute all scripts lo_import-TABLENAME.sh after setting the environment variables PGDATABASE and optionally PGHOST, PGPORT, PGUSER, etc. if they do not correspond to the default values for libpq. You might also execute manually a VACUUM FULL on the table to remove the bloat created by the table update. Limitation: the table must have a primary key, it is used to set the WHERE clause to update the Oid column after the large object import. Importing BLOB using this second method (--lo_import) is very slow so it should be reserved to rows where the BLOB > 1GB for all other rows use the option --blob_to_lo. To filter the rows you can use the WHERE configuration directive in ora2pg.conf. SUPPORT Author / Maintainer Gilles Darold Please report any bugs, patches, help, etc. to . Feature request If you need new features let me know at . This help a lot to develop a better/useful tool. How to contribute ? Any contribution to build a better tool is welcome, you just have to send me your ideas, features request or patches and there will be applied. LICENSE Copyright (c) 2000-2025 Gilles Darold - All rights reserved. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see < http://www.gnu.org/licenses/ >. ACKNOWLEDGEMENT I must thanks a lot all the great contributors, see changelog for all acknowledgments. ora2pg-25.0/changelog000066400000000000000000013630651500113072400145130ustar00rootroot000000000000002025 04 20 - v25.0 This major release fix several issues reported since last release and adds some new features and improvements. * Add multiple assessment report format output at once. Thanks to jcarnu for the patch. * Support more Oracle to PostgrSQL exception mapping. * Allow overriding of PG_SUPPORTS_* settings as they are set in the configuration file. PG_VERSION will have no effect in this case. Thanks to Pavel Stehule for the feature request. * Add multiple report format output at once. Thanks to Jean-Christophe Arnu for the patch. * Add parsing of ALTER statement from file for the QUERY action. * Add support for oracle_fdw COPY using CSV format * Add parallel export of each partition. Until now all partitions of a partitioned table was exported serialy in a single process. * Allow online data migration to continue if the destination table does not exist when ON_ERROR_STOP is disabled. Thanks to chetan2211 for the feature request. * Tables data export is now done using the current SCN to have the same snapshot of data between multiprocess export. * Add replacement of USERENV call with MODULE by current_setting with application_name. * Add support for data movement using psql copy selecting over oracle_fdw - Type INSERT (when FDW_SERVER is set) preserves prior behaviour of both INSERT and COPY - Type COPY (when FDW_SERVER is set) uses a new mode of using the `psql` with `\copy` with `TO PROGRAM...FROM STDIN BINARY...BINARY` Thanks to Martin Nash for the patch. * Add support for oracle_fdw in combination with psql "\copy" and server-side COPY using BINARY stream for data movement. Brings both local and server-side oracle_fdw binary copy into one branch. Control over which mode is used is provided via ORACLE_FDW_COPY_MODE configuration, which defaults to "local". Thanks to Martin Nash for the patch. * Convert DBMS_SQL cursor/parsing/execute simple form to PostgreSQL dynamic query execution. Thanks to tanguydelignieresaccenture for the patch with some regex improvement by me to handle comments. * Add replacement of the MOD operator. Thanks to ec-digit-dbecoe for the report. * Add some additional objects/difficulties assessment migration scores: - WHEN OTHER THEN exception clause that hide the real problem during migration. - 'SSSSS' and 'J' for to_char() that could need rewrites. * Add SCRIPT action to process sqlplus scripts as a whole, not line per line. * Change default settings in config file generated by the --init_project option with: - PG_NUMERIC_TYPE 0 - NULL_EQUAL_EMPTY 1 New options and configuration directives: * Add --no_clean_comment option to not remove comments in source file before parsing. With huge DDL file with comments, it could take a very long time. * Add -O | --options used to override any configuration parameter, it can be used multiple time. Syntax: -O "PARAM1_NAME=value" -O "PARAM2_NAME=value" or -O "PARAM1_NAME=value|PARAM2_NAME=value" * Add option --no_start_scn to force Ora2Pg to not use a SCN to export data unless --snc is used. By default the current SCN is used to export data from all tables. * Add ORACLE_FDW_COPY_MODE configuration directive. When using Ora2Pg COPY with oracle_fdw it is possible to use two different modes: 1) "local", which uses psql on the host running Ora2Pg for the "TO" binary stream; 2) "server", which uses PostgreSQL server-side COPY for the "TO" binary stream. Both modes use psql for the "FROM STDIN BINARY". However, "local" runs the psql "FROM STDIN BINARY" on host Ora2Pg is run from, whereas "server" runs the psql "FROM STDIN BINARY" on the PostgreSQL server. "local" mode should work on any PostgreSQL-based system, including managed offerings, which are not expected to support use of "server" mode due to permissions. The default is "local" as this is compatible with more configurations. * Add ORACLE_FDW_COPY_FORMAT configuration directive. When using Ora2Pg COPY with oracle_fdw it is possible to use either BINARY or CSV data format. BINARY provides better performance, however, requires exact data type matching between the FDW and destination table. CSV provides greater flexibiliity with respect to data type matching: if the FDW and destination data types are functionally-compatible the columns can be copied. The default is "binary". Bug fixes: - Fix always returns row count from stats, even if --count_rows flag is used. Thanks to alavrent2022 for the report. - Fix multiprocess with TEST_DATA action. Thanks to tanguydelignieresaccenture for the report. - Fix another if condition issue. Thanks to skm9380 for the report. - Fix assignment sign in UPDATE statements. Thanks to ec-digit-dbecoe for the report. - Fix mysql # comments. Thanks to ec-digit-dbecoe fir the report. - Fix enum column with value "inf" that is expanded to the word "Infinity". Thanks to nbromage for the report. - Fix MySQL labels not converted to PostgreSQL syntax. Thanks to ec-digit-dbecoex for the report. - Fix MySQL DECLARE declaration after BEGIN clause. Thanks to ec-digit-dbecoe for the report. - Fix error Invalid Identifier error for DBMS_LOB.GETLENGTH() by checking to user privilege. Thanks to tanguydelignieresaccenture for the patch. - Fix TO_DATE translation when a NLS_setting is present. Thanks to anguydelignieresaccenture for the patch. Fix export type TEST reporting 0 for PostgreSQL functions count if using default PACKAGE_AS_SCHEMA. Thanks to tanguydelignieresaccenture for the patch. - Use double quote instead of single quote in ora2pg command to fix an issue on windows with ora2pg_scanner. Thanks to ec-digit-dbecoe for the patch. - Fix TEST_DATA foreign table ordering consistency. Thanks to tanguydelignieresaccenture. - Fix ordering of Oracle data for TEST_DATA export. Thanks to tanguydelignieresaccenture for the patch. - Handling Group By and Having Clause inside subqueries (HAVING Clause misplacement in Ora2Pg). Thanks to newtora2pg for the patch. - Remove import of tmpdir function of Perl File::Spec module, it fails on Windows. - Limit search for spatial type in column to the first line. Thanks to Amit Kumar for the report. - Fix case where rewrite of listagg to string_agg is not done. Thanks to bbellsct for the report. - Apply reserved word quoting in PG query to for data validation. Thanks to mmay9CO for the report. - Prevent full scan when looking for spatial srid and dim. Thanks to Amit Kumar for the report. - Fix output format detection condition - Fix double declaration of variable - Append CREATE SCHEMA only if CREATE_SCHEMA is enabled for SEQUENCE export. Thanks to Brian Hayden for the report. - Fix schema name for procedures/functions when PG_SCHEMA is set. Thanks to mgole001 for the report. - Fix data export filename for partition when RENAME_PARTITION is enabled. Thanks to Simon Pane for the report. - Fix replacement of SYSTIMESTAMP by statement_timestamp() instead of CURRENT_TIMESTAMP. Thanks to dstinit for the report. - Fix methode import when using File::Spec. Thanks to niteshn24 for the report. - Apply MOFIDY_TYPE rewriting after REPLACE_AS_BOOLEAN. Thanks to jstmx for the report. - Fix inner query IN keyword missing after converting. Thanks to rajatjha28 for the report. - Fix export of procedure with out parameter when PG_version < 11. Thanks to Pavel Stehule for the report. - Remove rewrite of numeric operation with TRUNC() to use interval, too much false positive. Thanks to Pavel Stehule for the report. - Force binmode when output is compressed with bzip2. Thanks to Vladimir Roganov for the report. - Fix eval() error detection. Thanks to Vladimir Roganov for the report. - Fix incorrect translation from OUT to INOUT param - Handling Multiline COMMENT in Ora2Pg. - Handling Referencing in Triggers. - Handling Timestamp Datatype. - Handling XMLELEMENT Conversion. - [MSSQL] Add parsing of a DDL file to migrate tablesi with indexes and constraints. The GO keyword that ends a statements must be replaced by a semi-colon (;) and all brackets must be removed from the source file before. - Fix parsing of foreign keys with input file. Thanks to raulcejas04 for the report. - Fix schema for trigger function when read from file. - Remove double quote on custom type when readind table definition from a file. - Fix use of REPLACE_TABLES and REPLACE_COLS with input files. - Fix export of procedure to function when pg_supports_procedure is off by removing the unwanted extra_param parameter. Force pg_supports_outparams to off when pg_supports_procedure is off. Thanks to Pavel Stehule for the report. - Remove goldengate suplemental table logging - Fix pg_supports_ifexists with change in previous commit - Keep schema in function/procedure name when it is read from file an we are not in PACKAGE action. - Fix infinit loop in NLS_SORT replacement. - Avoid duplicate name in foreign keys constraints - Fix override of any Ora2Pg configuration directive from command line for complex values like REPLACE_TABLES. - Remove sinh, cosh and tanh from unsupported oracle function list - Fix parsing of stored procedure from file. - Fix if condition in export_schema.ps1. Thanks to Robin Pringle for the patch. - Fix replacement by PERFORM - Add "Schema Export Complete" and "Ora2Pg ending" message to provide enhancement documented in Issue #1806. Thanks to Simon Pane for the patch. - Quote PG_USER in ORACLE_FDW user mapping. Thanks to Simon Pane for the patch. - Remove PASSWORD from keywork list - Make sure that the column alias for trim() is not prefixed by the schema. - Fix TRUNCATE TABLE command when the table name needed to be quoted - Fix column list in COPY statements, columns was doubled. - Fix port setting for FDW server. Thanks to Aymen Zaiter for the report. - Added ORACLE_FDW_COPY_FORMAT: binary or csv - [mysql] Replace json_extrat() with json_extrat_path. Thanks to mgole001 for the report. - Remove table alias from column alias after TRIM function. Thanks to Simon Pane for the patch. - Fix validation of nullable boolean columns. Thanks to Eckhart Worner for the patch. - Fix regular expression for function-based indexes in validation. Thanks to Eckhart Worner for the patch. - Fix regression in NULL_EQUAL_EMPTY feature. Thanks to Eric Delanoe for the report. - Fix export of foreign keys for PG <= 10 - Fix rename of partitions when REPLACE_TABLES is used on the parent table and when PREFIX_PARTITION is enabled. - Set environment variable PGPASSWORD for ORACLE_FDW_COPY_FORMAT and remove PGPASSWORD from OFBC psql commands. Thanks to Martin Nash for the patch. Use of PGPASSWORD results in the password being exposed to anyone with host access. Switching to relying on .pgpass, which can be located in a custom location for Ora2Pg by setting PGPASSFILE in the session running Ora2Pg, avoids password exposure. - Update to use explicit column list for OFBC. Thanks to Martin Nash for the patch. - Fix conversion of data format for TO_TIMESTAMP function. Thanks to Priyanshi Gupta for the report. - Second fix for bug migrating data in json column. Thanks to mat-efluid for the report. - Fix migration of data from CLOB to jsonb isung COPY mode to preserve json escaping. Thanks to Thomas Herzog for the report. - Fix quoting for index creation on reserved keyword. Thanks to moonbeamglitterblossom for the report. - Fix case where values was transformed as boolean when when a table has the same name as a data type. Thanks to twiti7 for the report. - Fix use of TRANSFORM_VALUE when a function is used. Thanks to Thomas Herzog to the report - Fix for subquery where clause issue - Fix incorrect "unsupported partition type" warning for oracle_fdw copy. Thanks to Martin Nash for the patch. 2024 03 29 - v24.3 This release fix several issues reported since last release and adds some new features and improvements. * Add option control the "prefetch" used by oracle_fdw COPY/INSERT Prior to this change Ora2Pg uses the default "prefetch" of oracle_fdw, which at the time of writing is 50. Allowing this to be controlled by an Ora2Pg configuration/option gives the option of increased performance at the cost of some additional memory on the PostgreSQL side. Thanks to Martin Nash for the patch. * Modify the behavior of triggers export with EXPORT_INVALID. It used to apply to ENABLED or DISABLED triggers instead of real VALID or INVALID triggers. Export of INVALID triggers will be controlled by EXPORT_INVALID like others objects like functions, packages, etc. This mean that disabled triggers that are valid will be exported by default now, this was not the case before. Thanks to dcgadmin for the feature request. * Add new configuration directive PGTT_NOSUPERUSER. By default the pgtt extension is loaded using the superuser privilege when EXPORT_GTT is activated. Enabled it if you run the SQL scripts generated using a non superuser user. It will use: LOAD '$libdir/plugins/pgtt'; instead of default: LOAD 'pgtt'; Thanks to Simon Martin for the feature request. Here is the full list of changes and acknowledgements: - Fix Inf replacement that must only be done with numeric datatype. Thanks to gael-efluid for the report. - Fix some replacement of OUTER JOIN (+). Thanks to Carens Kurniawan Wijaya for the report. - Fix schema filter with test function count. Thanks to dcgadmin for the report. - Fix TEST count objects when a table name is modified. Thanks to korolan for the report. - Fix issue with multi style comments. Thanks to newtglobal.com for the patch. - Fix documentation about EXPORT_INVALID to precise that it also concern disabled triggers. Thanks to dcgadmin for the report. - Fix missing FOR EACH clause in trigger export after a regression introduced by commit fb6b0ad. Thanks to Carens Kurniawan Wijaya for the report. - Fix data export for table with a geometry column. Thanks to ruralqiu for the report. - Revert changes introduced by commit fc7008c, for some obscurs Oracle reasons the DBA_SDO_GEOM_METADATA doesn't always exist. Always uses ALL_SDO_GEOM_METADATA instead. Thanks to ruralqiu and Pierre3939 for the report. 2024 03 07 - v24.2 This release fix several issues reported since last release and adds some new features and improvements. * Allow DATA_EXPORT_ORDER to take a filename at value to be able to give a custom table order. The file must contain the ordered list of the tables. One table per line in upper case for Oracle. Thanks to DataCloudGaze for the feature request. * Add progress bar when --oracle_speed is used to avoid waiting for the whole data export ends. * Add replacement of the BITAND function by the & operator * Add option -f, --format to set the output format for the reports. It can be html or json. Default to html. Thanks to mgole001 for the feature request. * Add automatic addition of the partition key to the primary key. Prefix all columns with the alias in the query to get data from Oracle. * Add information about MSSQL masked columns in SHOW_COLUMN. * Add information about columnstore and compression on MSCSQL tables with action SHOW_TABLE. * Add new configuration directive PARTITION_BY_REFERENCE to defined how to export Oracle partition by reference. Possible values are none, duplicate or the number of hash partitions to create. Value 'none' mean no translation and export of partition by reference like before. Value 'duplicate' will duplicate the referenced column in the partitioned table and apply the same partitioning from the referenced table to the partitioned table. If the value is a number, the table will be partitioned with the HASH method using the value as the modulo. For example if you set it to 4 it will create 4 HASH partitions. Default is none to not export the partitions by reference definition. Here is the full list of changes and acknowledgements: - Fix default values in MSSQL function declaration and missing END keyword. - Fix parsing of MSSQL function with a single query. Thanks to Saravanan Newt for the report. - Fix negative Oracle decimal when comparing data. Thanks to es99-24 for the report. - Fix typos in documentation. Thanks to Simon Martin for the report. - Avoid doubling the NAME keyword in the XMLELEMENT() function. - Remove clause "REFERENCING OLD AS OLD NEW AS NEW" that is useless and throw an error in PostgreSQL - Fix DSN for MSSQL in autogenerated configuration file using --init_project. - Fix parsing of REFERENCING clause in trigger. Thanks to Carens Kurniawan Wijaya for the report. - Fix case where ALL_TAB_COLUMNS and ALL_SDO_GEOM_METADATA was used instead of USER_TAB_COLUMNS. Thanks to rvanouter for the report. - Fix double quoting in partition by reference where clause. - Disallow setting of PARTITION_BY_REFERENCE to duplicate when FDW_SERVER is set. - Fix trigger export with duplicate FOR EACH clause when a REFERENCING clause is present. Thanks to Carens Kurniawan Wijaya for the report. - Fix typo in variable name in ora2pg_scanner - Fix export of default partition for MySQL. Thanks to Priyanshi Gupta for the report. - Fix TEST_DATA action with issues on foreign server and import foreign schema declaration. Thanks to Florent Jardin for the report. - Fix export of NOT NULL constraint with column replacement. Thanks to Florent Jardin for the report. - Prevent reading file ora2pg_stdout_locker when it does not exist. Thanks to Florent Jardin for the report. - Fix translation of timestamp with precision < 6. Thanks to Andrei Briukhov for the report. - Fix json_arrayagg with returning clause, type was not translated. - Fix data export for table partitioned by reference with duplicate method. The query to extract data on Oracle side add the duplicated column and perform the join with the referenced table using the FK definition. - Fix partition by LIST export with a useless cast to text. Thanks to Priyanshi Gupta for the report. - Added closing curly bracket for object details. Thanks to andreas42 for the patch. - Added quotes around value of "human days cost" and "migration level". Thanks to andreas42 for the patch. - Format object type detail output as JSON array. Thanks to andreas42 for the patch. - Fix variable declaration in previous commit - Fix MSSQL table export with nonexistent column auto_created in version bellow 2017. Thanks to Florent Jardin for the report. - Use ADD CONSTRAINT syntax to specify name of primary key. Thanks to Martin Karlgren for the patch. - Skip unwanted work on PG database when option --oracle_speed is enabled, especially drop of constraints. Thanks to John Tian for the report. - Remove trailing ); from primary key statements when reading from file. Thanks to Martin Karlgren for the patch. - Fix option in CREATE USER MAPPING for export of DBLINK. - Fix translation of MSSQL floating point datatype to use float(n) notation. - Exclude from assessment objects in Oracle bin. - Add Oracle package HTP and HTF to migration assessment. - Fix MSSQL foreign key export with multiple columns referenced. - Fix MSSQL export of unique constraints that was merging columns of several unique constraints for the same table into a single constraint. - Fix case of MSSQL datetime default value 0 that must be converted to '1900-01-01 00:00:00' - Add rewrite of MSSQL getutcdate() function. - Fix MSSQL index type and add compression + columnstore information - Add clause IF NOT EXIST to all CREATE EXTENSION calls - Fix MSSQL bit data migration through tds_fdw, it is already exported as boolean by the fdw. - Fix duplicated indexes in MSSQL export. - Add export of MSSQL indexes with columns included (CREATE INDEX+INCLUDE). 2023 09 08 - v24.1 This release fix several issues reported since last release and adds some new features and improvements. - Replace "set feedback off" by "\set QUIET on;" and "set pagesize 0" with "\pset pager off". Thanks to Martin Gerhardy for the suggestion. - Always add package name into search_path of packages functions. Thanks to janopha for the report. - Allow to specify a password file to set PG_PWD at PostgreSQL database connection. If the specified file exists on the system, Ora2Pg will read the first line to get the password at each call of send_to_pgdb(). It can be useful in some situation where the password change during data migration. Thanks to Marius Hope for the feature request. - Added option --dump_as_json and fixed some json output errors. Thanks to Martin Gerhardy for the patch. Here is the full list of changes and acknowledgements: - Fix schema prefixing of type created in packages stored procedures. Thanks to janopha for the report. - Fix perl function get_schema_condition() to use quote_ident() in generated SQL filters. Thanks to franxav06 for the report. - Attempt to better test the not null constraint count in Oracle. - Fix regression with PSQL_RELATIVE_PATH. Thanks to Ryan Taylor for the report. - Do not add the partition key to PK if DISABLE_PARTITION is enabled. Thanks to pavel-moskotin-db for the report. - Filter list of indexes on name instead of the GENERATED column because we are missing the ones that have been created automatically by the Automatic Indexing feature of Oracle 19c. Thanks to Franck Pachot for the report. - Fix incompleteness in GRANT action, add grant usage on schema to owner and users. Thanks to elexus for the report. - Apply missing --blob_to_lo to import_all.sh script. - Fix regression in MySQL hash partitions export. Thanks to Shubham Dabriwala for the report. - Fix Oracle INTERVAL data export with negative value. Thanks to shubham-yb for the report. - Fix double count of not null constraints for TEST action. Thanks to Simon Pane for the patch. - Fix replacement of Oracle sys_refcursor in function return type. - Fix replacement of SQL script setting from Oracle. - Make scripts executable. Thanks to Martin Gerhardy for the patch. - Exclude data pump SYS_EXPORT_SCHEMA_.* tables from Oracle export. - ora2pg: use env based shebang for perl. Thanks to Martin Gerhardy for the patch. - Add unsupported clause message for PRAGMA AUTONOMOUS_TRANSACTION when it is not rewritten by Ora2Pg. Thanks to Martin Gerhardy for the patch. - Allow schema specific definitions of partitioning columns. Thanks to Martin Gerhardy for the patch. - Fixed invalid variable name in read_grant_from_file. Thanks to Martin Gerhardy for the patch. - Fix not double quoted column in alter set not null column statement. Thanks to leonteq-reisg for the patch. - Bug fixes and special treatment for Types with body. Thanks to Martin Gerhardy for the patch. - Avoid redundant definition of the version. Thanks to Martin Gerhardy for the patch. - Fix a MySQL exception when the column type is ENUM for version < 5.7. Thanks to Code-UV for the report. 2023 07 05 - v24.0 This major release adds support to migration of SQL Server database to PostgreSQL. It also fixes several issues reported since past height months and adds some new features and improvements. * Enable the use of ALLOW/EXCLUDE directive with SHOW_* reports and throw a fatal error if global filters in ALLOW/EXCLUDE are set. * Add replacement of DBMS_LOCK.SLEEP with pg_sleep * Split estimate cost details per function/procedure/and package function. * Add cmin, cmax, ctid to reserved keywords list. * Add cost for presence of ADD CONSTRAINT in PLSQL code. It needs constraint name stability. * Add CLOB_AS_BLOB configuration directive to treat CLOB as BLOB when exporting data. When enabled Ora2Pg will apply same behavior on CLOB than BLOB with BLOB_LIMIT setting. This could be useful if you have large CLOB data. Enabled by default. Thanks to Omar Mebarki for the patch. * Allow COPY and TABLE type to use the NULLIF construct. Thanks to Luke Davies for the patch. * Add new SEQUENCE_VALUES export type to export DDL to set the last values of sequences from current Oracle database last values like the following statements: ALTER SEQUENCE departments_seq START WITH 290; Thanks to sergey grinko for the feature request. * Add replacement of Oracle variable : varname into PG :'varname'. * Add SQL Server migration to Ora2Pg. Most of the SQL Server objects are supported as well as data export. Translation of the TSQL stored procedures to plpgsql is complicated because of the lack of statement separator in TSQL but as usual Ora2Pg is doing is best to do as much work as possible. Migration assessment is also possible with SQL Server database. There is some dedicated configuration directives added to ora2Pg.conf. * Add support to MySQL PARTITION BY KEY() with a translation to HASH partitioned table using the PK/UK definition of the table or the columns specified in the KEY() clause. Thanks to Shubham Dabriwala for the report. * Make EXPORT_INVALID configuration directive works with TRIGGER export. Until now disabled triggers were not exported, setting EXPORT_INVALID to 1 will force the export of disabled triggers. Thanks to chetank-yb for the report. * Add support of MySQL generated default value on update. For example: CREATE TABLE t1 ( dt DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ); Ora2Pg will translate this syntax into a trigger on the table to force the value of the column on an update event. Thanks to heysky for the report. * Add translation of ST_GEOMETRY data type to PostGis geometry datatype. * Replace ROWNUM in target list with a "row_number() over ()" clause. Thanks to Rui Pereira for the report. New configuration directives: * Add configuration directive ST_GEOMETRYTYPE_FUNCTION to be able to set the function to use to extract the geometry type from a ST_Geometry column. Default: ST_GeometryType, example it should be set to sde.ST_GeometryType for ArcSDE. Thanks to Albert88373 for the report. * Add four new configuration directive to be able to change or prefix the functions used to extract information from ST_Geometry object and values. - ST_SRID_FUNCTION: Oracle function to use to extract the srid from ST_Geometry meta information. Default: ST_SRID, for example it should be set to sde.st_srid for ArcSDE. - ST_DIMENSION_FUNCTION: Oracle function to use to extract the dimension from ST_Geometry meta information. Default: ST_DIMENSION, for example it should be set to sde.st_dimention for ArcSDE. - ST_ASBINARY_FUNCTION: Oracle function to used to convert an ST_Geometry value into WKB format. Default: ST_ASBINARY, for example it should be set to sde.st_asbinary for ArcSDE. - ST_ASTEXT_FUNCTION: Oracle function to used to convert an ST_Geometry value into WKT format. Default: ST_ASTEXT, for example it should be set to sde.st_astext for ArcSDE. Thanks to Albert88373 for the report. * Add INSERT_ON_CONFLICT configuration directive. When enabled this instruct Ora2Pg to add an ON CONFLICT DO NOTHING clause to all INSERT statements generated for this type of data export. Thanks to Clemens Rieder for the feature request. Backward compatibility: * Change the behavior of CASE_INSENSITIVE_SEARCH to allow the use of a collation instead of the citext extension. To disable the feature the value none can be used. If the migration is not MSSQL this feature is disabled. * Remove PREFIX_PARTITION configuration directive, it is now replaced by the RENAME_PARTITION directive. Previous behavior was to construct the partition name from the table name, the partition name and the sub partition name if any. The problem is that we often reach the max length for an object name and this leads to duplicate partition name. Now, when RENAME_PARTITION is enabled the partition tables will be renamed following rules: _part where "pos" is the partition number. For subpartition this is: _part_subpart If this is partition/subpartition default: _part_default _part_subpart_default This change will break backward comaptibilty, if PREFIX_PARTITION is still set, it will simply enable RENAME_PARTITION. * Set START value to MINVALUE when a sequence is cycled and that the START value is upper that MAXVALUE. Thanks to Shane Borden for the report. Here is the full list of changes and acknowledgements: - Fix MODIFY_STRUCT that was not working with MySQL. Thanks to Code-UV for the report. - Fix license string in Makefile.PL. Thanks to RodRaen for the report. - Do not remove non alphanumeric character in index name. Thanks to gwidt for the report. - Reorder trigger event when the update of column is not the last one. Thanks to tayalarun1 for the report. - Fix export of MySQL function containing special characters and white spaces in names. Thanks to Shubham Dabriwala for the report. - Fix grant export for partitions. Thanks to elexus for the report. - Add some other transformation for sqlplus/psql scripts. - Remove comma as possible separator for values in DEFINED_PK, it was preventing the use of a function with multiple parameters. - Fix export of geometry tables when PG_SCHEMA is set. - Add rewriting of some sqlplus settings to psql settings. - Fix TABLESPACE export for partitioned tables. Thanks to elexus for the report. - Fix for Issue #1637. Thanks to Simon Pane for the patch. - Fix typo in --init_project directories tree generation for sequences values. - Fix alias in view target list for function call without alias defined in MySQL export. Thanks to Shubham Dabriwala for the report. - Fix Mysql procedure export when a datatype with precision is used in parameter list. Thanks to Shubham Dabriwala for the report. - Fix collation on string default values. Thanks to Shubham Dabriwalafor the report. - Exclude recycle bin object from ALL_TAB_COLUMNS lookup. Thanks to Dave Betterton for the report. - Fix data types translation (TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT) for MySQL table export. Thanks to Shubham Dabriwala for the report. - Do not export synonym destination table with table_owner when EXPORT_SCHEMA is disabled. Thanks to Priyanshi Gupta for the report. - Fully qualify calls to get_sequence_last_values() when PG_SCHEMA is set. Thanks to Marius Hope for the report. - Fix regression on exporting view as table when VIEW_AS_TABLE contains regexp. Thanks to Neil Bower for the report. - Fix missing execution of initial command statements at start of TEST_DATA action and on both side, those applying to source and destination. Thanks to Petter Jacobsen for the report. - Fix script to get sequence last value with TEST action. Thanks to franxav06 for the patch. - Prepend PERFORM before call to DBMS_OUTPUT.* when USE_ORAFCE is enabled. - Disable USE_ORAFCE when export type is SHOW_REPORT. - Extending the enhancement in Pull Request #1621 to the Oracle_FDW user mapping. Thanks to Simon Pane for the patch. - Changed prefix string to "DIFF:" in test report. Thanks to Simon Pane for the patch. - Fix cases where %ROWCOUNT was not correctly replaced. Thanks to Rui Pereira for the report. - Fix parsing of ORACLE_DSN when creating foreign server in COPY mode. Thanks to Luke Davies for the report. - Fix for Issue #1622, #1627. Thanks to Simon Pane for the patch. - Fix index creation with DESC order in COPY action when DROP_INDEXES is enabled. Thanks to Luke Davies for the report. - Fix for Issue #1610, #1612, #1617 and #1381. Thanks to Simon Pane for the patch. - Fix typo in sqlnet.ora name (was sqlnet.or). Thanks to Martin Nash for the patch. - Fix data export, REPLACE_QUERY was not applied. Thanks to Bachev Constantin for the report. - Fix call to replace_sys_context(). - Fix timestamp(n) data type translation. - Remove use of column GENERATION_EXPRESSION for MySQL version < 5.7.0. Thanks to Hans Choi for the report. - Fix conversion of DATE datatype to timestamp(0) instead of timestamp. Thanks to Akhil Reddy for the report. - Add NVARCHAR/NCHAR defaut convertion data types to DATA_TYPE configuration directive in ora2pg.conf comments. Thanks to Akhil Reddy for the report. - Rename method _get_partitions_type function into _get_partitions_list. - Fix synonym export when no schema information is available. - Fix support of REFERENCING clause in triggers. - Fix partition output file renaming with new RENAME_PARTITION directive. Thanks to Rahul Barigidad for the report. - Fix export of the ROWNUM clause when there is a variable. - Fix sprintf placeholders in geometry queries. - Fix some others issues with row count report. - Fix row count with destination schema and when the PostgreSQL table doesn't exist. Thanks to bizen-ya for the report. - Fix tests comparison with the different settings of EXPORT_SCHEMA, SCHEMA and PG_SCHEMA. Thanks to Marius Hope and bizen-ya for the report. - Fix St_AsText() call for MySQL data extraction. - Add column count comparison for MySQL - Export multi column partition by list as an expression with concat operator. Multi column partition by list is not supported by PostgreSQL. - Fix creation of non existant indexes on partition. Thanks to Shubham Dabriwala for the report. - Fix MySQL function export when there is no BEGIN clause. Thanks to Shubham Dabriwala for the report. - Fix MySQL export of unsigned numeric. Thanks to Shubham Dabriwala for the report. - Fix MySQL output with wrong synthax for JOIN without ON clause. Thanks to Shubham Dabriwala for the report. - Fix virtual column export. Thanks to Rafal Hollins for the report. - Fix index creation on partition with no columns for MySQL export. Thanks to Shubham Dabriwala for the report. - Fix export of MySQL auto_increment when PG_INTEGER_TYPE is disabled. Thanks to Shubham Dabriwala for the report. - Fix MySQL subpartition export. Thanks to Sanyam Singhal for the report. - Move any INTO clause in CONNECT BY query to the final SELECT on the resulting CTE. Thanks to taptarap for the report. - Fix translation of MySQL curtime() function in default values. Thanks to Shubham Dabriwala for the report. - Fix possible "Nested quantifiers in regex" error when exporting package with package name containing regex special characters. Thanks to durandm70 for the report. - Fix documentation about use of unique key for ORACLE_COPY. - Fix extra comma at end of a CHECK contraint. Thanks to Shubham Dabriwala for the report. - Always add DROP TYPE statements with package export even if DROP_IF_EXISTS is not enabled. Thanks to Rui Pereira for the report. - Fix default value of simple dot in MySQL export. Thanks to Shubham Dabriwala for the report. - Fix regression in data type translation after fix on unsigned numeric type. Thanks to Shubham Dabriwala for the report. 2022 10 08 - v23.2 This release fix several issues reported since past height months and adds some new features and improvements. * Add export of MySQL KEY and LINEAR KEY partitioning, translated as HASH partitioning. * Allow export of object with dollar sign is his name. * Add export of CHECK constraints for MySQL >= 8.0. * Add Functional/Expression indexes export from MYSQL. * Add export of MySQL virtual column. Thanks to Shubham Dabriwala for the * Remove scale and precision of a numeric if the scale is higher than the precision. PostgreSQL does not support decimal/numeric attributes where the scale is explicitly higher than the precision. * Add command line option --drop_if_exists to add statement to drop objects before creation if it exists. It corresponds to the DROP_IF_EXISTS * Add option -C | --cdc_file to be able to change the name of the default file used to store/read SCN per table during export. Default is TABLES_SCN.log in the current directory. This is the file written by the --cdc_ready option. * Add multiprocess to count rows in PostgreSQL tables (TEST_COUNT) using -P command line option. * Add support to PostgreSQL 14 procedure with out parameters. * Set default PostgreSQL database target version to 14. New configuration directives: * Add configuration directive MVIEW_AS_TABLE and command line option --mview_as_table to set which materialized view to export as table. By default none. Value must be a list of materialized view name or regexp separated by space or comma. If the object name is a materialized view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. * Add configuration variable FDW_IMPORT_SCHEMA to rename the schema where foreign tables for data migration will be created. If you use several instances of ora2pg for data migration through the foreign data wrapper, you might need to change the name of the schema for each instance. Default: ora2pg_fdw_import * Add TRANSFORM_VALUE configuration directive to apply an expression when retrieving data from Oracle. For example: TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), ' ')] to replace all Oracle char(0) in a string by a space character. * Add EXCLUDE_COLUMNS configuration directive. Unlike MODIFY_STRUCT that is used to redefine a table structure, this directive allow to specify a list of columns per table that must be excluded from the export. For example: EXCLUDE_COLUMNS T1(nocol1,nocol2) T2(nocol1,nocol2) * Add new configuration directive EXPORT_GTT to export Oracle Global Temporary Table using syntax recognized by the pgtt extension. For more information see https://github.com/darold/pgtt Default is to not export global temporary table as they are not supported natively by PostgreSQL. * Add new configuration option NO_EXCLUDED_TABLE. By default Ora2Pg exclude from export some Oracle "garbage" tables that should never be part of an export. This behavior generates a lot of REGEXP_LIKE expressions which are slowing down the export when looking at tables. To disable this behavior enable this directive, you will have to exclude or clean up later by yourself the unwanted tables. The regexp used to exclude the table are defined in the array @EXCLUDED_TABLES in lib/Ora2Pg.pm. Note this is behavior is independent to the EXCLUDE configuration directive. Backward compatibility: * Force rewrite of all invalid date starting with zero year 0000 into 1970-01-01 when it is a default value and NULL for data. Old behavior was to only replace 0000-00-00 date. * Until now there was a lot of untranslated call to TRUNC(date) because Ora2Pg is unable to detect that the parameter is a date or a number. The problem is that Oracle has TRUNC(number) too and Ora2Pg try to not apply the transformation if there is a doubt. In most of the migration have met very few TRUNC(number) so now all call to TRUNC() will be converted to date_trunc(). There must be false positive rewrite but this should be far less work than the actual situation. Here is the full list of changes and acknowledgements: - Fixed PostgreSQL "relation not found error" in _dump_fdw_table(), PostgreSQL search_path was not being used. Thanks to James Schriever for the patch. - Fix year and month quoting as reserved words when they are used as aliases. Thanks to duursma for the report. - Fix conversion of to_number(substr(...)) when PG substr() return empty string where Oracle return NULL which make the conversion to numeric fail. The fix using (nullif(substr(...), )::numeric) only concern TABLE export. Thanks to Menelaos Perdikeas for the report. - Add export of MySQL KEY and LINEAR KEY partitioning, now exported as HASH partitioning. Thanks to Sanyam Singhal for the report. - Remove $ sign from characters that require object name quoting. - Fix export of objects with the $ sign in the name. Thanks to yano-rxa and duursma for the report. - Prevent translation of EXEC when used as alias. Thanks to Rui Pereira for the report. - Fix MySQL enum data type export with regression introduced by commit 24a476. Thanks to Shivansh Gahlot for the report. - Rename ORACLE_FDW_TRANSFORM to TRANSFORM_VALUE to apply in all case an expression when retrieving data from Oracle. For example: TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), ' ')] to replace all Oracle char(0) in a string by a space character. - Fix add_month() translation failing with some use cases. Thanks to duursma for the report. - Add export of CHECK constraints for MySQL >= 8.0. Thanks to Rahul Barigidad for the report. - Fix MySQL unsigned numeric data type conversion. Thanks to Rahul Barigidad for the report. - Add Functional/Expression indexes fail while exporting from MYSQL. Thanks to Shubham Dabriwala for the report. - Fix export of descending indexes for MySQL. Thanks to Shubham Dabriwala for the report. - Force MySQL auto increment sequence to start at 1 when value is 0. Thanks to Rahul Barigidad for the report. - Fix conversion of MySQL decimal(p,s) to keep the original datatype instead of use of real or double. - Add export of MySQL virtual column. Thanks to Shubham Dabriwala for the report. - Fix export of MySQL function with return clause only. Thanks to Shubham Dabriwala for the report. - Remove scale and precision of a numeric if the scale is higher than the precision. PostgreSQL does not support decimal/numeric attributes where the scale is explicitly higher than the precision. Thanks to Rahul Barigidad for the report. - Fix export of comment for views. Thanks to gh-k-murata for the report. - Add command line option --drop_if_exists to add statement to drop objects before creation if tehy exists. It corresponds to the DROP_IF_EXISTS configuration directive. Thanks to Yoni Sade for the feature request. - Add option --mview_as_table to documentation. - Add configuration directive MVIEW_AS_TABLE and command line option --mview_as_table to set which materialized view to export as table. By default none. Value must be a list of materialized view name or regexp separated by space or comma. If the object name is a materialized view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. - Disable EXPORT_GTT when export type is not TABLE. Thanks to gh-k-murata for the report. - Fix generated external servers wrongly placed in a schema. Thanks to duursma for the report. - Add configuration variable FDW_IMPORT_SCHEMA to rename the schema where foreign tables for data migration will be created. If you use several instances of ora2pg for data migration through the foreign data wrapper, you might need to change the name of the schema for each instance. Default to ora2pg_fdw_import. Thanks to James Schriever for the feature request. - Fix wrong conversion of rownum clause when a subquery is used. Thanks to Rui Pereira for the report. - Escape comma and backslashes in BFILE data export. Thanks to duursma for the patch. - Fix possible infinite loop in Oracle outer join parsing. Thanks a lot to yano-rxa for the report. - Remove privileges default settings on views exported from SYNONYMs - Add support for Rectangle geometry type. Thanks to duursma for the patch. - Fix double replacement of IS NULL/IS NOT NULL when NULL_EQUAL_EMPTY is enabled. - Add CHECK not null only constraints to not null constraint count. - Fix CHECK NOT NULL only constraints that was not exported by generating NOT NULL constraints instead. They are exclude from the count of CHECK constraint as suggested by Florent Jardin but a count difference persist for NOT NULL constraints. - Fix TYPE export when SCHEMA and PRESERVE_CASE have different values. Thanks to Florent Jardin for the report. - Fix custom exception replacement. Thanks to Rui Pereira for the report. - Fix Collection and Polygon geometry INTERNAL export. Thanks to duursma for the patch. - Fix export of efile with parenthesis. Thanks to duursma for the report. - Fix wrong column indices used in spatial index extraction. Thanks to duursma for the report. - Fix call of ST_GeomFromText() with WKT export. Thanks to duursma for the report. - Disable USE_LOB_LOCATOR with WKT geometry export type, ST_GeomFromText and SDO_UTIL.TO_WKTGEOMETRY functions return a CLOB instead of a geometry. Thanks to duursma for the report. - Fix INTERNAL conversion uses the srid from the object instead of the meta-data. Thanks to duursma for the report. - Fix regression in data export when REPLACE_AS_BOOLEAN is set. Thanks to Juri Berlanda for the report. - Fix call to procedure using dblink. Thanks to Rui Pereira for the report. - Keep untouched call to DBMS_OUTPUT functions if USE_ORAFCE is enabled. Thanks to Sanyam Singhal for the report. - Partial fix for MySQL subpartitioning export. - Fix partitions export for MySQL. Thanks to Sanyam Singhal for the report. - Fix generation of export_all.sh following the operating system. - Add information of use of PARALLEL_TABLES with COPY, INSERT and TEST_DATA actions. It is also useful with TEST, TEST_COUNT, and SHOW_TABLE if --count_rows is used for real row count. - Prevent calling real rows count twice with TEST action, and allow it for the SHOW_TABLE action. - Handle count errors when single process. - Move row count wait for all child die to the right place - Fix rewrite of nested replace() functions in CHECK constraint. Thanks to Menelaos Perdikeas for the report. - Fix call of procedures with out parameters when it is not declared in a package. Thanks to taptarap for the report. - Some minor code improvement. Thanks to Markus Elfring for the patch. - Set encoding to read configuration file to utf8. - Remove useless multiple semi-colon after END of a function. - Fix conversion of regexp_replace() by always appending the 'g' modifier. Thanks to Rui Pereira for the report. - Fix synonym detection to avoid listing public synonym when no schema is specified. Thanks to Dilan Salinda for the report. - Fix regexp error with multi-line comment in default value declaration. Thanks to taptarap for the report. - Add missing sub-partition key in partitioned table primary key. Thanks to downvoteit for the report. - Replace all invalid date starting with zero year 0000- to 1970-01-01 when it is a default value and NULL for data. Old behavior was to only replace 0000-00-00 date. Thanks to duursma for the report. - Enclose \i path to data file beween quote to fix import of table with space in their name. - Add PARTITION to the list of reserved work and fix custom keywords list from ORA_RESERVED_WORDS that was not applied. Thanks to markhooper99 for the report. - Add LOAD of pgtt extension before creating global temporary table with TABLE :export. Thanks to duursma for the report. - Fix case where package names should be lower cased. Thanks to Sergey Petrov for the patch. - Cover more case where ALTER ... OWNER TO should not be generated. - Fix case where ALTER ... OWNER TO should not be generated when a view as table definition was not exported. - Fix sub-partition unique and primary keys that lacks columns part of the partition key. Thanks to downvoteit for the report. - Path for function_per_file are mixed case enabled now. Thanks to Sergey Petrov for the patch. - Fix AUTOINCREMENT script to set last value to sequences for serial an identity column for PG version < 12. Thanks to Jaouad Bouras for the report. - Fix detection of ENUM data type for MySQL - Fix issue when exporting table with a geometry column. The search for the SDO_GTYPE need a FROM clause with a FQDN table when the connection user is not the same as the table schema. Thanks to Argo64 for the report. - Rewrite numeric operation with ADD_MONTH(), LAST_DAY() and TRUNC() to use interval. Thanks to duursma for the report. - Fix rewrite of CONNECT BY in cursors and just after a BEGIN. Thanks to taptarap for the report. - Add partition keys to unique index on partitioned table. Thanks to downvoteit for the report. - Fix case where global variable are tested against NULL. Thanks to duursma for the report. - Fix remove of %ROWTYPE in function argument and returned data type. Add regression test. Thanks to Eric Delanoe for the report. - Fix case clause in autoincrement parameters. Thanks to jbouras for the report. - Fix typo in ORACLE_FDW_TRANSFORM example. - Fix progress bar output in quiet mode. Thanks to Sanyam Singhal for the report. - Fix error Can't locate object method "gzclose" via package "IO::File". Thanks to Sanyam Singhal for the report. - Fix cases where translation of function with out parameter was not done correctly. - Fix translation of function with out parameter that returns a value. For example the following Oracle function: CREATE FUNCTION foo(a int, OUT b int) RETURN boolean IS BEGIN b := a; RETURN true; END; is now translated by adding an extra out parameter for the return value: CREATE OR REPLACE FUNCTION foo (a integer, OUT b integer, OUT extra_param boolean) RETURNS record AS $body$ BEGIN b := a; extra_param := true; RETURN; END; $body$ LANGUAGE PLPGSQL STABLE; Thanks to Akhil Reddy for the report. - Fix undefined database connection handle. Thanks to Alexander for the report - Fix case preservation for row count in Oracle side with TEST_COUNT action.. Thanks to Veka for the report. - Only generate the Powershell script when we are running on a Windows operating system. - Fix #1400 and generate PowerShell script "export_schema.ps1". Thanks to moh-hassan for the report. - Fix rewriting assignment of a global variable using SELECT INTO. Thanks to duursma for the report. - Fix partition export for MySQL. Thanks to Sanyam Singhal for the report. - Apply WHERE clause to FDW data export. - Fix useless ST geometry parsing. Thanks to jieguolove for the report. - Replace backslash with slash in BFILE filename when destination data type is text or efile. - Fix RAW(16)/RAW(32) data export when MOFDIFY_TYPE is used on the column. Thanks to Sergey Evseev for the report. - Fix ST_SRID() call. Thanks to jieguolove for the report. - Skip table data export when the table has no column defined. This was generating a fatal error. - Fix untranslated function returned data type when there was a comment just after. The comment is removed. Thanks to taptarap for the report. - Fix other fetching all-column-all-table properties for every table. Thanks to Sergey Petrov for the report. - Fix fetching all-column-all-table properties for every table. Thanks to Sergey Petrov for the report. - Remove any comments between RETURN and returned type to not break parsing. Thanks to taptarap for the report. - Fix global variables in the DECLARE section are not replaced if used with a package name. Thanks to taptarap for the report. - Remove renaming of dist configuration file under Windows OS. Thanks to Julien Monticolo and ohamed Hassan for the report. - Fix remaining data export query failure. Thanks to Sung Woo Chang for the report. - Fix data export, query to retrieve data was broken since change for GTT. - Set function as VOLATILE when there is CALL in the body. - Add support to PG14 procedure out parameters. Thanks to Rui Pereira for the feature request. - Fix missing parenthesis in index column expression with input file. - Fix missing END keyword after embedded CASE clause in a package function. Thanks to taptarap for the report. - Fix conversion of dbms_lob.substr() where second and third parameters must be inverted. Thanks to taptarap for the report. - Fix an other case of wronf NOT NULL detection from input file. - Fix detection of NOT NULL constraint in input file. - Do not quit on error "Undefined subroutine &Ora2Pg::ReadLine", just continue to be able to leverage an Oracle Wallet (SEPS) when no Oracle user and password are provided. If you want to use the interactive mode to type the username and password at command line you must install the Perl package Term::ReadKey before. Thanks to Simon Pane for the report. - Fix partitioning by LIST, only the first value of a list was exported. Thanks to Sergey Grinko for the report. - Fix quoting of DEFAULT NULL. Thanks to Veka for the report. - Fix unwanted multiple CALL keywords. Thanks to taptarap for the report. - Add assessment counter for FND_* packages. - Fix LONG RAW export as bytea in COPY mode. Thanks to Helena Adiduyulmus for the report. - Add new configuration option NO_EXCLUDED_TABLE. By default Ora2Pg exclude from export some Oracle "garbage" tables that should never be part of an export. This behavior generates a lot of REGEXP_LIKE expressions which are slowing down the export when looking at tables. To disable this behavior enable this directive, you will have to exclude or clean up later by yourself the unwanted tables. The regexp used to exclude tables are defined in the array @EXCLUDED_TABLES in lib/Ora2Pg.pm This behavior is independent to the EXCLUDE configuration directive. Thanks to Peter Humaj for the feature request. - Replace all remaining CURSORNAME%NOTFOUND with NOT FOUND - Change translation to SYSDATE from LOCALTIMESTAMP to statement_timestamp() in non PL/SQL code. - Prevent append of SECURITY DEFINER when a procedure execute transaction control statements (ex: COMMIT). When defined with this clause an error is thrown. Thanks to Suman Michael for the report. 2022 02 10 - v23.1 This release fix several issues reported since past four months and adds some new major features and improvements. * Add use of greatest/least functions from new version of Orafce when required to return NULL on NULL input like Oracle. * ALLOW and EXCLUDE configuration values can now be read from a file. Use -a filename or -e filename to specify the list of tables that need to be filtered. This is useful if you have a lot of table to filter. * Add possibility to use of System Change Number (SCN) for data export or data validation by providing a specific SCN. It can be set at command line using the -S or --scn option. You can give a specific SCN or if you want to use the current SCN at first connection time set the value to 'current'. To use this last case the connection user must have the role "SELECT ANY DICTIONARY" or "SELECT_CATALOG_ROLE", the current SCN is looked at the v$database view. Example of use: ora2pg -c ora2pg.conf -t COPY --scn 16605281 This adds the following clause to the query used to retrieve data for example: AS OF SCN 16605281 You can also use th --scn option to use the Oracle flashback capability by specifying a timestamp expression instead of a SCN. For example: ora2pg -c ora2pg.conf -t COPY --scn "TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS')" This will add the following clause to the query used to retrieve data: AS OF TIMESTAMP TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS') or for example to only retrieve yesterday's data: ora2pg -c ora2pg.conf -t COPY --scn "SYSDATE - 1" * Add json output format to migration assessment. Thanks to Ted Yu for the patch. * Add new TO_CHAR_NOTIMEZONE configuration directive to remove any timezone information into the format part of the TO_CHAR() function. Disabled by default. Thanks to Eric Delanoe for the report. Note that the new default setting breaks backward compatibility, old behavior was to always remove the timezone part. * Add new configuration directive FORCE_IDENTITY_BIGINT. Usually identity column must be bigint to correspond to an auto increment sequence so Ora2Pg always force it to be a bigint. If, for any reason you want Ora2Pg to respect the DATA_TYPE you have set for identity column then disable this directive. * Add command line option --lo_import. By default Ora2Pg imports Oracle BLOB as bytea, the destination column is created using the bytea data type. If you want to use large object instead of bytea, just add the --blob_to_lo option to the ora2pg command. It will create the destination column as data type Oid and will save the BLOB as a large object using the lo_from_bytea() function. The Oid returned by the call to lo_from_bytea() is inserted in the destination column instead of a bytea. Because of the use of the function this option can only be used with actions SHOW_COLUMN, TABLE and INSERT. Action COPY is not allowed. If you want to use COPY or have huge size BLOB ( > 1GB) than can not be imported using lo_from_bytea() you can add option --lo_import to the ora2pg command. This will allow to import data in two passes: 1) Export data using COPY or INSERT will set the Oid destination column for BLOB to value 0 and save the BLOB value into a dedicated file. It will also create a Shell script to import the BLOB files into the database using psql command \lo_import and to update the table Oid column to the returned large object Oid. The script is named lo_import-TABLENAME.sh 2) Execute all scripts lo_import-TABLENAME.sh after setting the environment variables PGDATABASE and optionally PGHOST, PGPORT, PGUSER, etc. if they do not correspond to the default values for libpq. You might also execute manually a VACUUM FULL on the table to remove the bloat created by the table update. Limitation: the table must have a primary key, it is used to set the WHERE clause to update the Oid column after the large object import. Importing BLOB using this second method (--lo_import) is very slow so it should be reserved to rows where the BLOB > 1GB for all other rows use the option --blob_to_lo. To filter the rows you can use the WHERE configuration directive in ora2pg.conf. * Add command line option --cdc_ready to use current SCN per table when exporting data and register them into a file named TABLES_SCN.log This can be used for Change Data Capture (CDC) tools. * Allow to export only invalid objects when EXPORT_INVALID is set to 2 * Disable per partition data export when a WHERE clause is define on the partitioned table or that a global WHERE clause is defined. Backward compatibility: Ora2Pg used to removr any timezone information from the TO_CHAR() format function. To recover this behavior set TO_CHAR_NOTIMEZONE to 1 in ora2pg.conf Complete list of changes: - Replace PERFORM by CALL when the stored procedure is a procedure. Thanks to Rui Pereira for the report. - Fix open cursor translation when using is in the query but not as keyword. Thanks to taptarap for the report. - Fix replacement of global variables in DECLARE section. Thanks to taptarap for the report. - Fix missing suffix in function name with autonomous transaction when export schema was enabled and fix revoke and owner to wrapper function. Thanks to Sergey Grinko for the report. - Fix export of type declaration in packages without body. Thanks to Sergey Grinko. - Fix column name duplicates when exporting data of partition. Thanks to Sergey Grinko for the report. - Fix BLOB export with INSERT mode, call decode() was missing. - Fix applying of DEFAULT_PARALLELISM_DEGREE hint that was not working anymore for a long time. Thanks to Marcel Pils for the patch. - Update documentation about PARALLEL_TABLES and view export. Thanks to xinferum for the report. - Fix unwanted quote escaping in global variable constant. Thanks to sergey grinko for the report. - Fix export of global variable when there is function in the default value. - Fix end of statements in last merged PR. - Add json output format to migration assessment. Thanks to Ted Yu for the patch. - Fix parsing of package when a comment follow the AS keyword. Thanks to Eric Delanoe for the report. - Adapt MAXVALUE for identity columns if the datatype has been changed to integer. - Fix a regression on data validation introduced with commit to fix data export of virtual column. - Fix Can't locate object method is_pk via package Ora2Pg error - Exclude unique keys using expression to validate data. - Fix ORDER BY clause for data validation. - Fix error on open pragma when encoding is not set. - Fix a regression in data export of virtual column. Thanks to Code-UV and IgorM12 for the report. - Fix a second regression with empty column name in target list to retrieve data. - Fix PG version to enable virtual column. - Fix binmode when it is set to raw or locale to not call encoding() in open pragma. - Fix regression in export view as table. Thanks to Sebastian Albert for the report. - Update Copyright year. - Quote tables names when necessary during TEST action. - Fix undefined call to auto_set_encoding(). - Add test count of column per table and add output of the PG table struct modified to be used with MODIFY_STRUCT. - Fix handling of PRESERVE_CASE with update au sequences values - Fix handling of PRESERVE_CASE with TEST_DATA - Fix unwanted replacement of sysdate operation to epoch. Thanks to taptarap for the report. - Remove extra END clause at end of package function when a space or a comment was present. Thanks to taptarap for the report. - Fix missing import of module Encode. Thanks to Menelaos Perdikeas for the report. - Fix case where data type defined in function was not exported when EXPORT_SCHEMA was enabled. Thanks to Eric Bourlon for the report. - Fix missing EXECUTE on OPEN CURSOR statements. Thanks to taptarap for the report. - Fix missing declaration of min() function in Oracle.pm. Thanks to nicscanna for the report. - Fix SYSDATE subtract of seconds instead of days - Fix PERFORM replacement in CTE. Thanks to taptarap for the report. - Fix wrong stored procedure code conversion when use types named with "default" and broken decode to case translation. Thanks to taptarap for the report. - Add missing import of FTS indexes in script import_all.sh. Thanks to vijaynsheth for the report. - Fix another procedure parsing with return. Thanks to Eric Bourlon for the report. - Fix case where parenthesis are not added to index creation. - Add creation of the uuid extension when it is used. - Add HTML report of tables and columns with name > 63 characters. - Add report of DBMS_ERROR and Quartz Scheduler tables found. - Add mark (date?) on columns of DATE data type in Oracle to check if it should be translated into date instead of default timestamp. - SHOW_COLUMN: mark column data type with (numeric?) when it is a NUMBER without precision. - SHOW_TABLE+SHOW_COLUMN: Add mark of tables and columns name > 63 characters - Fix translation of TYPE ... AS TABLE OF ... - Fix parsing of function call in check constraints. Thanks to Menelaos Perdikeas for the report. - Fix missing data export file for partitioned tables when TRUNCATE_TABLE was disabled. Thanks to Menelaos Perdikeas for the report. - Fix named parameter inserted in procedure call with inout parameters. Thanks to Rui Pereira for the report. - Fix unwanted quoting of index columns clause when there is an operation. Thanks to Menelaos Perdikeas for the report. - Move comment in procedure parameters before the BEGIN. Thanks to Eric Bourlon for the report. - Fix parsing of FOR CUSOR followed by a parenthesis. Thanks to Eric Bourlon for the report. - Fix parsing of TYPE ... IS REF CUSOR declaration in procedures. Thanks to Eric Bourlon for the report. - Add replacement of SDO_CS.TRANSFORM into ST_Transform. Thanks to mukesh3388 for the report. - Add missing table namer to index renaming. - Create a function for index renaming for code reuse. - Fix support translation of type VARRAY from store procedure. Thanks to Eric Bourlon for the report. - Fix conversion of SQL%ROWCOUNT when part of a string concatenation. Thanks to boubou191911 for the report. - Remove other non alphanumeric character from index name. Thanks to Menelaos Perdikeas for the report. - Fix date formatting when error is logged with INSERT failure. Thanks to xinjirufen for the report. - Remove possible comma from index renaming. - Fix drop of indexes with renaming when there is a function call. Thanks to Menelaos Perdikeas for the report. - Fix empty geometry type since the move of ORA2PG_SDO_GTYPE into lib/Ora2Pg/Oracle.pm - Move most of the Oracle specific code to a dedicated Perl library lib/Ora2Pg/Oracle.pm with the same functions as lib/Ora2Pg/MySQL.pm This will help to maintain and extend Ora2Pg to other RDMS. There is still Oracle database related specific code in the main library but it will be also moved later. There should not be any regression or usage change with this huge patch. - Fix translation of type with not null clause. Thanks to Yasir1811 for the report. 2021 11 15 - v23.0 This release fix several issues reported since past five months and adds some new major features and improvements. * Add new option --blob_to_lo that can be used to export BLOB as large objects. It can only be used with action SHOW_COLUMN, TABLE and INSERT. When used with TABLE action, the BLOB column will be translated into oid PostgreSQL data type. When used with the INSERT export action BLOB data will be store as large object in the pg_largeobjects table and the oid referencing this large object will be stored in the main table instead of a bytea. It is not possible to use oid with COPY because this feature use function lo_from_bytea() that stores the large object in the external table and returns the oid. This feature works with or without the use of oracle_fdw to import the data and option -J can be used to improve the speed of the INSERT import provide that there is a numeric unique key on the table. Thanks to rodiq for the feature request. * Add command line option -W | --where clause to set the WHERE clauses to apply to the Oracle query to retrieve data. It can be used multiple time. It will override the WHERE configuration directive if there is a global WHERE clause or the same table WHERE clause definition. Otherwise the clause will be appended. * Add data validation feature consisting in comparing data retrieved from a foreign table pointing to the source Oracle table and a local PostgreSQL table resulting from the data export. By default Ora2Pg will extract 10000 rows from both side, you can change this value using DATA_VALIDATION_ROWS. When it is set to zero all rows of the tables will be compared. Data validation requires that the table has a primary key or unique index and that the key columns is not a LOB. Due to differences in sort behavior between Oracle and PostgreSQL, if the collation of unique key columns in PostgreSQL is not 'C', the sort order of is different compared to Oracle. In this case the data validation will fail. Ora2Pg will stop comparing two tables after 10 errors, result is dumped to an output file named data_validation.log. * Add DATA_VALIDATION_ORDERING configuration directive enabled by default. Order of rows between both sides are different once the data have been modified. In this case data must be ordered using a primary key or a unique index, that mean that a table without such object can not be compared. If the validation is done just after data import in mode single process and without any data modification the validation can be done on all tables without any ordering. * Add DATA_VALIDATION_ERROR to stop validating data from a table after a certain amount of row mismatch. Default is to stop after 10 rows validation errors. * Allow multiprocess for TEST_DATA action to validate data import. Use -P or PARALLEL_TABLES to set the number of parallel tables checked. Output is now done to a file named data_validation.log saved in the current directory. * Add replacement of UTL_RAW.CAST_TO_RAW with encode(). * Add rewrite of XMLTYPE() with xmlparse(DOCUMENT convert_from(..., 'utf-8')). * Add VARCHAR_TO_TEXT configuration directive. By default VARCHAR2 without size constraint are tranlated into text PG data type. If you want to use varchar instead, disable this directive. * Add detection of XML function for migration assessment cost. * Add DBMS_RANDOM to the list of Oraclism handled by Orafce. * Add support to mysql_fdw foreign data wrapper to export data PostgreSQL tables. Thanks to Yoni Sade for the feature request. * Allow to transform all NUMBER(*,scale) to an other data type by a redefinition like NUMBER(*\,2):decimal in the DATA_TYPE configuration directive. Thanks to Florent Jardin for the patch. * Add information on how to use SSL encrypted connection to documentation. * Add TEST_COUNT action to just report the row count diff between Oracle and Backward compatibility changes: - Add FORCE_PLSQL_ENCODING configuration directive. By default Ora2Pg encode all functions code to ut8, this sometime can generate double encoding. To change this behavior, disable this configuration directive. Thanks to rynerisraid and lee-jongbeom for the report. - Change behavior regarding RAW columns. Now RAW(16) and RAW(32) columns or RAW columns with "SYS_GUID()" as default value are now automatically translated into uuid. Data will be automatically migrated as PostgreSQL uuid data type provided by the "uuid-ossp" extension. To recover the old behavior to export data as bytea whatever is the precision, the following must be set with DATA_TYPE configuration: RAW(16):bytea,RAW(32):bytea Complete list of changes: - Fix USE_LOB_LOCATOR handling. - Fix data validation using oracle_fdw where zero after decimal is not strip unlike with PG. - Apply MODIFY_STRUCT redefinition to test actions - Fix PG filter when DATA_VALIDATION_ORDERING is disabled - Apply RAW to uuid transformation for data validation - Apply boolean transformation for data validation - Do not export data for virtual column for PG >= 13. - Fix wrong replacement function with name including a regexp_* function in his name. Thanks to Rui Pereira for the report. - Remove comments in the from clause before rewrite outer join (+), the entire FROM clause will be rewritten and we don't know where to restore. - Fix export of columns information for data verification. - Fix TEST_VIEW for row count returned by views to exclude views created in extensions. - Fix comment on procedures - Fix translation of MySQL type UNSIGNED - Fix test count of indexes for MySQL database. - Fix test MySQL sequence count. - Do not display error messages when user and db is first checked in the import_all.sh script - Fix ordering of check constraints - Fix mysql table scan when table name is using reserved word. Thanks to Stanley Sung for the report. - Fix double BOTH keyword in TRIM function. Thanks to Rui Pereira for the report. - Fix aliases placed in a wrong way. Thanks to Rui Pereira for the report. - Fix parsing of procedure broken on keyword RETURN. Thanks to Pavithra Jayasankar. - Fix case where default partition is taken as a value. Thanks to Karsten Lenz for the report. - Fix conversion of NUMBER without precision in PL/SQL code to respect settings PG_NUMERIC_TYPE, PG_INTEGER_TYPE and DEFAULT_NUMERIC. Fix translation of INTEGER/BINARY_INTEGER that was wrongly exported as numeric. Thanks to Philippe Beaudoin for the report. - Documentation fix. Thanks to mperdikeas for the patch. - Fix case where SQL%ROWCOUNT was not replaced by GET DIAGNOSTIC. Thanks to Awdotia Romanowna for the report. - Fix quote of unique constraints name. Thanks to Veka for the report. - Fix looking at package function metadata when there is a huge amount of package. - Fix error when trying to remove temporary files. - Fix wrong translation of a call to a procedure with PRAGMA AUTONOMOUS TRANSACTION through dblink. Thanks to Rui Pereira for the report. - Remove schema name in front of index name. Thanks to Menelaos Perdikeas for the report. - Fix virtual column generated from an other column of the table (supported in PG 12). Thanks to Veka for the report. - Fix case of columns names in boolean transformation when oracle_fdw is used to export data. Thanks to veka for the report. - Remove extra parenthesis with sub query and TABLE function. Thanks to Rui Pereira for the report. - Fix WHERE clause not removed in ROWNUM replacement. Thanks to Rui Pereira for the report. - Exclude extensions tables from table test count. Thanks to Yoni Sade for the report. - Fix pg_attribute column adsrc removed in PG 12. Thank to Thorsten Hochreuter for the patch. - Fix unwanted aliases after row_number() over(). Thanks to Rui Pereira for the report. - Fix several spelling issues. Thanks to Florian Eckert for the patch. - Fix wrong condition to import constraints in import_all.sh. Thanks to Thorsten Hochreuter for the report. - Fix BITMAP_AS_GIN detection. Thanks to Nishanth Bejgam for the patch. - Fix parsing of views from file and add PASSWORD, KEY and REF to the list of reserved keywords. - Fix replacement of CURSOR ... IS when there is comment after IS. - Fix comment in auto generated file global_variables.conf - Fix XML data export that was transformed by the call to function extract(/).getClobVal(), it is now replaced by a direct call to getClobVal(). - Improve COPY FREEZE data export when FILE_PER_TABLE is enabled, the transactions are now managed per individual file and not following the main file. Thanks to Yoni Sade for the report. - Fix addition to UNLOGGED keyword on foreign table when exporting data using oracle_fdw. Thanks to Veka for the report. - Fix FK error when using TRUNCATE before data export with oracle_fdw. - Fix export of user defined type. Actually type definitions are extracted from ALL_SOURCE which contain the original CREATE TYPE and eventually all the ALTER TYPE commands. Previously those type as considered as not supported by Ora2Pg. - replace date(n) by timestamp. - Always remove the fqdn SYS schema before functions call. - Add report of GTT in SHOW_TABLE action. - Fix empty partition values for Oracle 9i. - Add creation of schema in user defined type export when EXPORT_SCHEMA is enabled to fix an error when the schema has not already been created. - Fix some wordings and exclude from export user defined type starting with SYS_PLSQL_ found in a 9i export. It looks that they are internal to PL/SQL code. - Exclude DBMS_SQL from the DBMS count in migration assessment when USE_ORAFCE is enabled. - Handle case where indexes name include the schema at create time - Fix PL/SQL numeric datatype conversion 2021 07 02 - v22.1 This is a maintenance release to extend the feature of data export through the oracle_fdw PostgreSQL extension to migration that use the public schema and do not preserve case. There is also some other fixes: - Fix compile_schema() call that breaks valid function based indexes by adding compile_all => FALSE to DBMS_UTILITY.compile_schema(). Thanks to Pawel Fengler for the patch. - Force foreign table for data export as readonly to avoid accidental write if import schema is not cleaned. - Fix data export to file not possible since last changes for oracle_fdw export. Thanks to Niels Jespersen for the report. 2021 06 26 - v22.0 This release fix several issues reported since past three months and adds some new features and improvements. I must thanks MigOps Inc who hire me to drive Oracle to PostgreSQL migrations and to develop Ora2Pg. It's been a long time that I was looking for such a company and it is an amazing gift for the 20 years of Ora2Pg. All improvements and new new features developed during my work at MigOps will be available in the public GitHub repository, here are the new ones. - Add export of data using oracle_fdw when FDW_SERVER is set and export type is COPY or INSERT. Multi-process using -P or -J is fully supported but option -j is useless in this case. Boolean transformation of some columns or data type is also supported. Actually, expect that it works just like data migration without oracle_fdw. This can improve the data migration speed from 30 to 40% especially for BLOB export. - Improve export performances with huge number of objects by avoiding join between Oracle catalog tables. - Set a maximum of assessment score for tables, indexes, sequences, partitions, global temporary table and synonym following the number of objects. - Add detection of XML functions to the assessment cost. - Allow to change the assessment cost unit value in the export_all.sh script when ora2pg is used with options --init_project and --cost_unit_value. - Remove pragma restrict_references from P/PSQL code, it is useless. - Add the oracle schema to search_path in SQL files generated and improve the migration assessment when USE_ORAFCE is enabled. - Apply ALLOW and EXCLUDED filtered stored procedures at package extraction level. Previous this patch there was no way to not export some package functions or to exclude them from assessment. - Add new tests to check sequences last values and number of identity columns in both side. - Apply ALLOW/EXCLUDE without object to table object by default in TEST action. New configuration directives: - Add ORACLE_FDW_TRANSFORM configuration directive to apply a transformation to a column when exporting data. Value must be a semicolon separated list of TABLE[COLUMN_NAME, ] For example to replace string 'Oracle' by 'PostgreSQL' in a varchar2 column use the following. ERROR_LOG[DBMS_TYPE:regexp_replace("DBMS_TYPE",'Oracle','PostgreSQL')] Thanks to MigOps for the patch. - Add DROP_IF_EXISTS configuration directive to add a statement "DROP IF EXISTS" before creating the object. Can be useful in an iterative work. Default is disabled. Thanks to dherzhau for the feature request. Backward compatibility: There is a backward compatibility issue with old configuration files where FDW_SERVER is set by default. This directive was not used when exporting data, this is not the case anymore as it instruct Ora2Pg to use the given foreign server to use oracle_fdw to migrate the data. Here is the full list of changes and acknowledgements: - Fix replacement of TO_CLOB() function, now it is just removed and the parenthesis are kept. Thanks to Rui Pereira for the report. - Fix incorrect detection of cursor on dynamic query. Thanks to Rui Pereira for the report. - Fix quoting column names with spaces and dots. Dots are replaced by underscore. Thanks to Veka for the report. - Fix one case where DEFINED_PKEY with PRESERVE_CASE was not handled correctly. Thanks to Veka for the report. - Fix quoting of reserved keywords in CREATE INDEX columns names. Thanks to Veka for the report. - Fix column name starting with number not quoted in COMMENT. Thanks to Veka for the report. - Fix addition of PERFORM on call to stored procedures not prefixed by the package name. Thanks to Rui Pereira fo the report. - Fix search of ora2pg_conf.dist under Windows instead of ora2pg.conf.dist when --init_project is used. Thanks to Julien Monticolo for the report. - Fix translation from file of check constraint when created on same column, only the last one was exported. Also shortened the prefix for constraint naming, ora2pg_ckey becomes o2pc, ora2pg_ukey is now o2pu and ora2pg_fkey is renamed into o2pf. Thanks to anvithaprabhu8 for the report. - Replace wildcard precision * for numeric by 38. - Fix incomplete listagg() conversion. Thanks to avandras for the report. - Fix potential problem in last_day conversion when USE_ORAFCE is off and a number is added or subtracted to the last day. Thanks to atlterry for the report. - Do not apply utf8 conversion of comments to input files. - Fix termination of last writer process when parallel and quiet mode are used together. Thanks to David Harper for the patch. - Remove precision in number of digit in timestamp microseconds when setting NLS_TIMESTAMP_FORMAT at session startup. Now use: ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS.FF' instead of '.FF6'. This was possibly the cause of sporadic error: ORA-03106: fatal two-task communication protocol error (DBD ERROR: OCIStmtFetch) Default is precision 6 so it may not change anything. Thanks to Abhijeet Bonde for the report. - Fix call to deprecated MySQL spatial function when version is after 5.7.6. Thanks to naveenjul29 for the report. - Fix false positive detection for nested table with MySQL export. - Fix sequence export read from file. - Add PG_VERSION to the documentation. Thanks to xinjirufen for the report. - Fix error report when preparing query to ALL_IND_COLUMNS IC. Thanks to ganeshakorde for the report. - ora2pg_scanner: Fix detection of service_name in DSN. - Fix error raise_application_error() with named parameters. - Fix replacement of out parameters in triggers. - Change sequence export result storage to hash instead of array. - Fix package export when there is a comment between PACKAGE BODY and the name of the package. - Fix Perl error "malformed utf-8 character in substitution" when there is character not in utf8 in the comment or constants. - Fix detection of MySQL FUNCTION vs PROCEDURE for version >= 5.5. Thanks to naveenjul29 for the report. - Exclude nested tables from the export as it is not supported and it always generate an error. A warning is raised. - Fix column case in check constraints when PRESERVE_CASE is enabled. - Fix search_path in direct PG data export when PG_SCHEMA is set. - Fix -Infinity insert for direct PG data export. - Fix drop indexes when PRESERVE_CASE is enabled. - Remove potential double affectation for function with out parameter. - Create immutable to_char function when used in an index. - Replace dmake by gmake on Windows installation instruction. Thanks to Julien Monticolo for the report. - Fix MySQL version conditions. Thanks to Christoph Berg for the report. - Fix HASH partitioning for duplicate WITH clause. - Fix tests when no schema are set to compare all objects in all schemes. Thanks to gp4git and dlc75 for the report. - Apply ALLOW/EXCLUDE without object to table object by default in TEST action. Thanks to Yony Sade for the feature request. - Add DROP_IF_EXISTS configuration directive. Thanks to dherzhau for the feature request. - Fix regression in removing %ROWTYPE from function parameters. Thanks to Eric Delanoe for the report. - Fix not adding default values to parameters when this is an OUT parameter. Thanks to Eric Delanoe for the report. - Fix ALL_DIRECTORIES call be using table name relative to USER_GRANTS. Thanks to Yoni Sade for the report. - Change all remaining call to static ALL_* tables to a call relative to USER_GRANTS. Thanks to Yoni Sade for the report. - Fix export or partitioned table with unsupported partitioning type like PARTITION BY REFERENCE. The table is created without partition and a warning it fired as well as the following message in the output file as a comment: -- Unsupported partition type, please check 2021 04 01 - v21.1 This release fix several issues reported since past six months and as usual adds some new features and improvements. * Now that Orafce 3.15.0 has a definition for the REGEXP_* function, makes the translation optional to USE_ORAFCE directive. * Add set application name in connection to Oracle/MySql/PostgreSQL. * Add translation of REGEXP_COUNT() and change assessment cost. * Rewrite the way REGEXP_LIKE() is translated into regexp_match to support modifiers. This rewrite also fix default behavior between Oracle and PostgreSQL. * Replace DBMS_LOB.GETLENGTH() by PostgreSQL octet_length() function. * Add types correspondences for VARCHAR2 and NVARCHAR2 in DATA_TYPE configuration directive. * Add autodetection and support of geometry type, srid and dimension for ArcGis geometries. * Add conversion of default value in function parameters. * Add -u | --unit option to ora2pg_scanner to be able to set the migration cost unit value globally. * Replace DBMS_LOB.SUBSTR() by SUBSTR() * Remove TO_CLOB() it is useless, manual cast could be necessary. * Replace IS JSON validation clause in CHECK constraints by (CASE WHEN $1::json IS NULL THEN true ELSE true END) When the code is invalid an error is fired. * DISTINCT and UNIQUE are synonym on Oracle Backward compatibility changes: - Force a column to be bigint if this is an identity column. Thanks to MigOps.com for the patch. - Fix EMPTY_LOB_NULL, enable/disable was inverted, keeping default to enabled. Take care that in old ora2pg.conf it is disabled so it will break backward compatibility with old configuration. - Replace NO_LOB_LOCATOR with USE_LOB_LOCATOR and NO_BLOB_EXPORT with ENABLE_BLOB_EXPORT to avoid confusion with double negative variable. Backward compatibility is preserved with a warning. - SRID for SDO_GEOMETRY export is now taken from the value not forced from the metadata table. Here is the full list of changes and acknowledgements: - Take Geometry SRID from the data and fallback to SRID defined in metadata when not found. Thanks to Sebastian Albert for the report. - Fix case where Ora2Pg temporary substitution of '' by placeholder was not restored. Thanks to MigOps.com for the patch. - Fix identity column export on unsupported Oracle 18c options. Thanks to MigOps.com for the patch. - Fix export of columns indexes created with single quote. Thanks to MigOps.com for the patch. - Fix replacement of keyword PROCEDURE by FUNCTION in constraints constants definition. Thanks to marie-joechahine for the report. - Replace IS JSON validation clause in CHECK constraints. Thanks to marie-joechahine for the report and MigOps.com for the patch. - Add support to ON OVERFLOW clause in LISTAGG replacement. Thanks to MigOps.com for the patch. - Fix incorrect handling of HAVING+GROUP BY rewriting. Thanks to MigOps.com for the patch. - Add replacement of TO_NCHAR by a cast to varchar. Thanks to MigOps.com for the patch. - Fix replacement of NOTFOUND when there is extra space or new line in the WHEN EXIT clause. Thanks to MigOps.com for the patch. - Fix a regression in NO_VIEW_ORDERING, it was not taken in account anymore. Thanks to RonJojn2 for the report. - Replace DATA_TYPE with DTD_IDENTIFIER in MySQL catalog queries for version prior 5.5.0. Thanks to zejeanmi for the report. - Fix import script to import sequences before tables. Thanks to MigOps.com for the patch. - Fix detail report of custom type in migration assessment. Thanks to MigOps.com for the patch. - Fix duplicate schema prefixed to SYNONYM. Thanks to dlc75 for the reports. - Replace NO_LOB_LOCATOR with USE_LOB_LOCATOR and NO_BLOB_EXPORT with ENABLE_BLOB_EXPORT to avoid confusion with double negative variable. Thanks to Rob Johnson for the report. - Fix some missing replacements of NVL and rewrite !=-1 into != -1. Thanks to MigOps.com for the patch. - Fix ROWNUM followed by + or - operator and when no aliases are provided. Thanks to MigOps.com for the patch. - Add DBSFWUSER to the list of user/schema exclusion. Thanks to MigOps.com for the patch. - Fix regexp to not append subquery aliases on JOIN clause. Thanks to Rui Pereira for the report. - Handle PRESERVE_CASE and EXPORT_SCHEMA in sequence name. Thanks to marie-joechahine for the report. - Add CREATE SCHEMA statement to sequence export when EXPORT_SCHEMA is enabled. Thanks to marie-joechahine for the report. - Fix duplicate index name on subpartition. Thanks to Philippe Beaudoin for the report. - Exclude sequences used for IDENTITY column (ISEQ$$_). Thanks to marie-joechahine for the report. - Fix parsing from file of CREATE SEQUENCE. Thanks to Rui Pereira for the report. - In export_all.sh script use the database owner provided if it is a superuser instead of postgres user. Thanks to jjune235 for the feature request. - Fix parsing of triggers when there is a CASE inside the code. Thanks to Rui Pereira for the report. - Add set application name in connection to Oracle/MySql/PostgreSQL. Thanks to Yoni Sade for the patch. - Fix double column alias when replacing ROWNUM. Thanks to Rui Pereira for the report. - Add translation of the REGEXP_COUNT function and change assessment cost. - Rewrite the way REGEXP_LIKE is translated into regexp_match to support modifiers. This rewrite also fix default behavior between Oracle and PostgreSQL. Thanks to otterrisk for the report. - Add IS JSON to assessment. Thanks to marie-joe Chahine for the report. - Fix multi-columns RANGE partitioning. Thanks to Philippe Beaudoin for the report. - Improve reordering columns. Sort by fieldsize first, if same size then it sorts by original position. Thanks to Sebastien Caunes for the patch. - Append partition's column to the primary key of the table as it must be part of the PK on PostgreSQL. Thanks to xinjirufen for the report. - Fix partition export where PRESERVE_CASE was applied to Oracle side. Thanks to schleb1309 for the report. - Fix trigger export with column restriction. Thanks to Sebastien Caunes for the report. - Update installation information. - Fix table reordering following data type. Thanks to Sebastien Caunes for the patch. - Fix incorrect variable name corresponding to DATA_EXPORT_ORDER making this directive inefficient. Thanks to Ron Johnson for the report. - Fix translation of check constraint when read from file - Fix EMPTY_LOB_NULL, enable/disable as inverted, keep default to enabled. Take care that in old ora2pg.conf it is disabled so it will break backward compatibility with old configuration. - Fix false positive detection of input filename is the same as output file. - Rename variables SCHEMA_ONLY, DATA_ONLY and CONSTRAINTS_ONLY in script import_all.sh to conform to their real use. Thanks to Sebastien Caunes for the report. - Fix comment detection breaking the package header parsing and global variable detection. - Fix ROWNUM detection for replacement by LIMIT - Fix escaping of psql command in configuration file comment and set default value for PG_VERSION to 12. - Replace precision by exactness in documentation. Thanks to Sebastien Caunes for the report. - Prevent reducing DATA_LIMIT when NO_BLOB_EXPORT is enabled. Thanks to Thomas Reiss for the report. - Fix geometry type detection. - Add autodetection of geometry type, srid and dimension for ArcGis geometries. Thanks to changmao01 for the feature request. - Fix call to ST_GeomFromText when no SRID is found. - Fix case where OVERRIDE SYSTEM VALUE clause could be added if PG version is < 10. Thanks to changmao01 for the report. - Fix unwanted call to internal GEOM library for ArcGis geometries. Thanks to changmao01 for the report. - Exclude schema SDE (ArGis) from export. Thanks to changmao01 for the report. - prevent looking twice to same custom data type definition. - Fix previous patch to catch SDO_GEOMETRY on lowercase regexp. - Limit detection of geometry data type to SDO_GEOMETRY. - Fix column name replacement in view definition. Thanks to Amit Sanghvi for the report. - Fix REPLACE_COLS parsing to allow space in column name. Thanks to Amit Sanghvi for the report. - Fix translation from file of triggers with WHEN clause. Thanks to Rui Pereira for the report. - Fix column name kept lowercase in the MOD() clause when -J is used. Thanks to Code-UV for the report. - Keep case of PG_SCHEMA definition when used in TEST action. - Fix data export for columns with custom data type. Thanks to Aymen Zaiter for the report. - Fix missing bracket with || operator in CREATE INDEX. Thanks to traxverlis for the report. - Fix export of single row unique function base index. Example: CREATE UNIQUE INDEX single_row_idx ON single_row ((1)); Thanks to unrandom123 for the report. - Update documentation about schemas used in TEST action. - Disable materialized view export with MySQL export it is not supported. Thanks to naveenjul29 for the report. - Fix table alias detection in Oracle (+) join rewrite. - Fix an infinite loop in Oracle (+) join rewrite when there is no table aliases and the table is prefixed by its schema. Thanks to Olivier Picavet for the report. - Fix MODIFY_STRUCT when column name need to be escaped. Thanks to helmichamsi10 for the report. - Fix empty PARTITION BY () clause. Thanks to Aymen Zaiter. - Fix export of global variable from package description when there is no package body. Thanks to naveenjul29 for the report. - Add package description export when dumping package source, previously only the package body was dump. This will allow to check global variables export. - Whilst working on the Reproducible Builds effort (https//reproducible-builds.org/) it appears that ora2pg could not be built reproducibly. Thanks to Chris Lamb for the patch. - Fix case of NUMBER(*,10) declaration. Oracle has a precision of 1 to 38 for numeric. Even if PostgreSQL allow a precision of 1000 use 38 to replace junk parameter. Thanks to xinjirufen for the report. - Add conversion of default value in function parameters, like syssdate rewriting for example. Thanks to unrandom123 for the report. - Fix a regression in data encoding when exporting data introduced in commit fa8e9de. Thanks to gp4git for the report. - Add debug information about the environment variables used before connecting to Oracle. - Fix case of duplicate between unique index and unique constraint with multiple columns. Thanks to gp4git. 2020 10 12 - v21.0 This release fix several issues reported since last release and adds several new features and improvements. * Add clause OVERRIDING SYSTEM VALUE to INSERT statements when the table has an IDENTITY column. * Considerably increase the speed to generate the report about the migration assessment, especially for database with huge number of objects. * Reduce time passed in the progress bar. Following the number of database objects we were spending too much time in refreshing the progress bar. * Add number of identity columns in migration assessment report. * Make assessment details report initially hidden using HTML5 tags
* Improve speed of BLOB/CLOB data export. Oracle recommends reading from and writing to a LOB in batches using a multiple of the LOB chunk size. This chunk size defaults to 8k (8192). Recent tests show that the best performances can be reach with higher value like 512K or 4Mb. * Add progress bar when --oracle_speed is use in single process mode. * Automatically activate USER_GRANTS when the connection user has no DBA privilege. A warning is displayed. * Complete port to Windows by using the Windows separator on stdout redirection into a file at ora2pg command line call and improve ora2pg_scanner port on Windows OS. * Add rewrite of MySQL JOIN with WHERE clause instead of ON. * Add MGDSYS (Oracle E-Business Suite) and APEX_040000 to the list of schemas excluded from the export. * Supply credentials interactively when a password is not defined in the configuration file. Need the installation of a new Perl module Term::ReadKey. * Add supports oracle connections "as sysdba" with username "/" and an empty password to connect to a local oracle instance. * Add translation of PRIVATE TEMPORARY TABLE from Oracle 18c into PostgreSQL basic temporary table, only the default behavior for on commit change. New command line options: * Add new command line option to ora2pg_scanner: -b | --binpath DIR to set the full path to directory where the ora2pg binary stays. Might be useful only on Windows OS. * Add -r | --relative command line option and PSQL_RELATIVE_PATH configuration directive. By default Ora2Pg use \i psql command to execute generated SQL files if you want to use a relative path following the script execution file enabling this option will use \ir. See psql help for more information. New configuration directives: * NO_VIEW_ORDERING: By default Ora2Pg try to order views to avoid error at import time with nested views. With a huge number of views this can take a very long time, you can bypass this ordering by enabling this directive. * NO_FUNCTION_METADATA Force Ora2Pg to not look for function declaration. Note that this will prevent Ora2Pg to rewrite function replacement call if needed. Do not enable it unless looking forward at function breaks other export. * LOB_CHUNK_SIZE See explanation in the new features and improvement list. * ALTERNATIVE_QUOTING_REGEXP To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals set the regexp with the text capture to use to extract the text part. For example with a variable declared as c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; the regexp to use must be: ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' ora2pg will use the $$ delimiter, with the example the result will be: c_sample varchar(100) := $$This doesn't work.$$; The value of this configuration directive can be a list of regexp separated by a semi colon. The capture part (between parenthesis) is mandatory in each regexp if you want to restore the string constant. Backward compatibility changes: - Default for NO_LOB_LOCATOR is now 1 to benefit from the LOB_CHUNK_SIZE performances gain. - Enable schema compilation (COMPILE_SCHEMA set to 1) by default to speed up DDL extraction. - Change the behavior of Ora2Pg with the parameters that follows a parameter with a default value. Ora2Pg used to change the order of the parameter's function to put all parameters with a default value at end of the list which need a function call rewrite. This have been abandoned now any parameter without default value after a parameter with a default value will be appended DEFAULT NULL. Here is the full list of changes and acknowledgements: - Fix unwanted references to PK/UK when DROP_INDEXES is enabled. - Fix comparison between function name in TEST report. - Fix duplicates on retrieving partitions information. - Improve SHOW_TABLE report about partitioned tables information. - Drop code about removing DEFAULT NULL in functions parameters. Thanks to chaluvadi286 for the report. - Fix two other case where materialized view can be listed in the table list. - Fix case where materialized view can be listed in the table list. Thanks to Thomas Reiss for the report. - Fix %ROWTYPE removing to be restricted to REF CURSOR. Thanks to jagmohankaintura-tl for the report. - Fix PG functions count when comparing Oracle functions count in TEST action. Remove useless -l option to import_all.sh auto generated script. - Fix PRESERVE_CASE on schema name for functions extracted from a package. - Fix search_path adding public default schema. - Apply PRESERVE_CASE to partition by involved columns. - Add IF EXIXTS to create schema to avoid error when import_all.sh is run several time. - Fix sort order of comment on columns for tables and views. - Fix warning about data export from nonexistent table resulting of index lookup on nested table. - Fix infinite loop in global variables package extraction. Thanks to Thomas Reiss for the report. - Fix global variables and packages export when comments are present in the package description. - Add information about XML_PRETTY size limit to 4000 - Fix column name in indexes when PRESERVE_CASE is enabled. Thanks to Julien traxverlis for the report. - Fix Top 10 of largest tables sort order. Thanks to Tom Vanzieleghem for the patch. - Fix duplicates between indexes and constraints. Thanks to sdpdb and Jon Betts for the report. - Fix SYSDATE replacement and possible infinite loop in SYSDATE parsing. Thanks to pbidault for the report. - Fix export of Oracle TEXT indexes with USE_UNACCENT disabled. Thanks to Eric Delanoe for the report. - Add new configuration directive ALTERNATIVE_QUOTING_REGEXP to support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals. Thanks to just-doit for the report. - Fix OF clause missing in update triggers. Thanks to just-doit for the report. - Fix IS NULL translation in WHERE clause of UPDATE statement. Thanks to Eric Delanoe for the report. - Remove DDL export of LOG indexes on materialized views. - Fix unexpected materialized view listed in table export. Thanks to jagmohankaintura-tl for the report. - Fix default values with single quote in create table DDL. Thanks to justdoit for the report. - Fix double quote in CREATE TRIGGER code and applying of preserve case on column name. - Supply credentials interactively when a password is not defined in configuration file. Thanks to rpeiremans for the patch. - Add supports oracle connections "as sysdba" with username "/" and an empty password to connect to a local oracle instance. Thanks to rpeiremans for the patch. - Fix documentation about materialized view export. - Fix export order of comments on columns. - Fix export of views comments when no schema is used for export and export schema is activated. - Fix cast in replacement with TO_NUMBER and TO_CHAR in indexes. Thanks to Kiran for the report. - Add MGDSYS (Oracle E-Business Suite) to the list of schemas excluded from the export. Thanks to naveenjul29 for the report. - Add more information about PG_DSN use. Thanks to Pepan7 for the report. - Update copyright year. - Fix regression where "SET client_encoding TO ..." was missing data file header. Thanks to Emmanuel Gaultier for the report. - Fix EDITABLE vs EDITIONABLE parsing. Thanks to Naveen Kumar for the report. - Fix typos in documentation. Thanks to swallow-life, ChrisYuan, Edward Betts, Jack Caperon and cavpollo for the patches. - Add OVERRIDING SYSTEM VALUE to INSERT statement when the table has an IDENTITY column. Thanks to Robin Windey for the report - Remove empty parenthesis of identity column options - Limit sequence/identity column value to bigint max - Add an example of DBD::Oracle DSN with 18c. - Fix parsing of identity column from file. Thanks to deepakp555 for the report. - Fix quoting of identifier when PRESERVE_CASE is enable and no particular schema is specified. Thanks to mkgrgis for the report. - Move setting of search_path before truncate table. Thanks to Michael Vitale for the report. - Add explanation about TEST and SIZE migration assessment values. - Mark XMLTYPE as having LOB locator. - Fix XMLTYPE columns that are exported as lob locator. Thanks to Tamas for the report. - Fix a problem of data export throughput that was slowing down all along the export when multiprocess for output was not used. Ora2Pg was forking a process for each chunk of data (see DATA_LIMIT) which is useless when write output is done on a single process (-j 1) and slow down the export. Thanks to markhooper99 and Tamas for reporting, testing and finding the source of the issue. - Fix progress bar in multiprocess mode, update was not displayed at each chunk of data processed. - Add internal debug information for progress bar. - Add debug information for SHOW_REPORT - Fix a long pending issue with custom data type export. Thanks to jhollandsworth for the patch. - Fix LOB data export with value changed to NULL when the CLOB value was 0. Thanks to jhollandsworth for the report. - Fix escape format issue with COPY and bytea. Thanks to Christoph Noel and dwbrock62 for the report. - Add LD_LIBRARY_PATH and PATH prerequisite to run ora2pg. - Fix use of the HIGH_VALUE column in partition listing with Oracle 9i. Thanks to Francisco Puga for the report. - Update the table row count logic to incorporate the PostgreSQL table FQN as established through the set_pg_relation_name routine. Thanks to Jacob Roberts for the patch. - Add the PostgreSQL FQN when printing the results in the TEST function. Thanks to Jacob Roberts for the patch. - Do not look forward function with the SHOW_* action - Fix BLOB export where \x was escaped. Thanks to Christophe Noel for the report. - Update Ora2Pg.pm to fix symbol in column name in create index statement. Thanks to kpoluektov for the patch. - Fix package function extraction when there is a start of comment (/*) in a constant string. Thanks to Tiago Anastacio for the report. - Fix type detection in package declaration. Thanks to Tiago Anastacio for the report. - Avoid displaying error ORA-22831 when exporting LOB. This error can appears when LOB chunk size is different from default 8192. The error has no incidence on the export so we can just ignore it. This patch also use DBD::Oracle ora_lob_chunk_size() method to gather chunk the chunk size of the LOB, fallback to 8192 if not available. Thanks to joedbadmin for the report. - Disable direct report of Oracle errors, all error should be handled at Ora2Pg level. - Fix MySQL data export with allow/exclude objects. Thanks to Manuel Pavy for the report. - Fix exclude/allow object feature in MySQL export that was not working since release 19.0. Thanks to Manuel Pavy for the report. - Add rewrite of MySQL JOIN with WHERE clause instead of ON. Thanks to Marc Rechte for the report. - Fix issue with custom type when multiprocess is used. - Fix progress bar on final total estimated data in multiprocess mode. - Fix ORACLE_HOME path in README.md. Thanks to Lubos Cisar for the patch. - Fix missing replacement with PERFORM in CASE ... WHEN statements. Thanks to Eric Delanoe for the report. - Fix duplicate ora2pg command in iteration. - Improve ora2pg_scanner port on Windows OS. Thanks to Marie Contencin for the report. - Add perl call to all ora2pg commands when the scanner is executed on Windows system as the shebang is not recognized. Thanks to Marie Contencin for the report. - Fix several issue with compressed output. Thanks to Bach Nga for the report. - Fix translation of CURSOR IS SELECT with a comment before the SELECT. Thanks to Izaak van Niekerk for the report. - Fix export of procedures as PostgreSQL procedures with version 11. - Add APEX_040000 to the schemas exclusion list. Thanks to Don Seiler for the report. - Fix possible unquoted default values. Thanks to Marc Rechte for the report. - Fix MySQL SET TRANSACTION clause when TRANSACTION is set to readonly or readwrite this is not supported so fall back in READ COMMITTED isolation level in this case. Thanks to Marc Rechte for the report. - Fix export of functions, column DATA_TYPE does not exists in table INFORMATION_SCHEMA.ROUTINES before MySQL 5.5.0. Replace it with column DTD_IDENTIFIER for prior version. Thanks to Marc Rechte for the report. - Fix double quote in CREATE TRIGGER code and applying of preserve case on column name. 2019 01 18 - v20.0 This release fix several issues reported during the last three months and adds several new features and improvement. The change of major version is related to backward compatibility break with the removed of most PG_SUPPORTS_* configuration directives and their replacement with the new PG_VERSION directive. New features and configuration directives in this release: * Add PG_VERSION configuration directive to set the PostgreSQL major version number of the target database. Ex: 9.6 or 10. Default is current major version at time of a new release. This replace the old PG_SUPPORTS_* configuration directives. * Removed all PG_SUPPORTS_* configuration directives minus PG_SUPPORTS_SUBSTR that is related to Redshift engine. * Export of BFILE as bytea is now done through a PL/SQL function to extract the content of a BFILE and generate a bytea data suitable for insert or copy into PostgreSQL. * Foreign keys that reference a partitioned table are no more exported. * Show table name on Oracle side during export using at connection time: DBMS_APPLICATION_INFO.SET_ACTION(table_name); * When the date format is ISO and the value is a constant the call to to_date() is removed and only the constant is preserved. For example: to_date(' 2013-04-01 00:00:00','SYYYY-MM-DD HH24:MI:SS') is replaced by a simple call to: ' 2013-04-01 00:00:00'. This rewrite is limited to PARTITION export type when directive PG_SUPPORTS_PARTITION is enabled. * Add DATA_EXPORT_ORDER configuration directive. By default data export order will be done by sorting on table name. If you have huge tables at end of alphabetic order and are using multiprocess, it can be better to set the sort order on size so that multiple small tables can be processed before the largest tables finish. In this case set this directive to size. Possible values are name and size. Note that export type SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or INSERT export type. * Add NO_BLOB_EXPORT configuration directive. Exporting BLOB could take time and you may want to export all data except the BLOB columns. In this case enable this directive and the BLOB columns will not be included into data export. The BLOB column must not have a NOT NULL constraint. Thanks to Ilya Vladimirovich for the * Add PREFIX_SUB_PARTITION to enable/disable sub-partitioning table prefixing in case of the partition names are a part of the sub- partition names. * Add special replacement for case of epoch syntax in Oracle: (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 is replaced by the PostgreSQL equivalent: (extract(epoch from now())) Here is the full list of changes and acknowledgements: - Export indexes and constraints on partitioned table with pg >= 11. - Fix incorrect replacement of NLS_SORT in indexes. - Bring back DISABLE_UNLOGGED feature. Thanks to Jean-Christophe Arnu for the patch - Fix CREATE SCHEMA statement that was not written to dump file. - Fix DBMS_APPLICATION_INFO.set_action() call, old Oracle version do not support named parameters. - Fix duplicate index name on partition. Thanks to buragaddapavan for the report. - Add support to new configuration directive PG_VERSION to control the behavior of Ora2Pg following PostgreSQL version. - Fix error in creation of default partition with PostgreSQL 10. Thanks to buragaddapavan for the report. - Fix missing export of single MAXVALUE partition, this will produce the following range partition: ... FOR VALUES FROM (MINVALUE) TO (MAXVALUE) Previous behavior was to not export partition as it is better to not partition the table at all. However it is declared in Oracle so it is better to export it to see what can be done. Thanks to buragaddapavan for the report. - Do not export foreign keys that reference a partitioned table. Remove NOT VALID on foreign keys defined on a partitioned table if present. Thanks to Denis Oleynikov for the report. - Fix export of BFILE as bytea. Ora2Pg now use a PL/SQL function to extract the content of a BFILE and generate a bytea data suitable for insert or copy into PostgreSQL. Thanks to RickyTR for the report. - Add TIMEZONE_REGION and TIMEZONE_ABBR to migration assessment, no direct equivalent in PostgreSQL. Remove NSLSORT not used in migration assessment. Thanks to buragaddapavan for the report. - Fix output of multiple export type specified in TYPE directive. - Rewrite and renaming of _get_sql_data() function into _get_sql_statements(). - Limit CURSOR weight in migration assessment to REF CURSOR only, other case are all covered. REF CURSOR might need a review to see if they need to be replaced with a SET OF RECORD. - Fix replacement of EMPTY_CLOB() or EMPTY_BLOB() with empty string when EMPTY_LOB_NULL is disabled and NULL when it is enabled. - Prefix output file with the export type in multiple export type mode, ex: sequence_output.sql or table_output.sql. Thanks to buragaddapavan for the report. - Fix export of data from an Oracle nested table. Thanks to rejo oommen for the report. - Removed cast to timestamp from partition range. Thanks to buragaddapavan and rejo-oommen for the report. - Fix partition default syntax. Thanks to rejo-oommen for the report. - Apply missing SYSUSERS schemas exclusion on columns and partition listing. Thanks to rejo-oommen for the report. - Add warning about parameter order change in output file. - Show table name on Oracle side during export using at connection time: DBMS_APPLICATION_INFO.SET_ACTION(table_name); Thanks to Denis Oleynikov for the feature request. - Report change in ORA_RESERVED_WORDS into documentation. - Add references in the keyword list of ORA_RESERVED_WORDS. - Fix the missing white space in some lines while creating import_all.sh file. Thanks to Fabiano for the patch. - Fix translation of infinity value for float. Thanks to Damien Trecu for the report. - Fix default value in timestamp column definition when a timezone is given. Thanks to buragaddapavan for the report. - Fix missing export of index and constraint in a partitioned table when DISABLE_PARTITION is enabled. Thanks to Denis Oleynikov for the report. - Prevent PARTITION BY when DISABLE_PARTITION is enabled. Thanks to Denis Oleynikov for the report. - Add DATA_EXPORT_ORDER configuration directive. By default data export order will be done by sorting on table name. If you have huge tables at end of alphabetic order and are using multiprocess, it can be better to set the sort order on size so that multiple small tables can be processed before the largest tables finish. In this case set this directive to size. Possible values are name and size. Note that export type SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or INSERT export type. Thanks to Guy Browne for the feature request. - Fix remove leading ':' on Oracle variable taking care of regex character class. Thanks to jselbach for the report. - Add NO_BLOB_EXPORT configuration directive. Exporting BLOB could take time and you may want to export all data except the BLOB columns. In this case enable this directive and the BLOB columns will not be included into data export. The BLOB column must not have a NOT NULL constraint. Thanks to Ilya Vladimirovich for the feature request. - Fix incorrect rewrote of the first custom type in a row. Thanks to Francesco Loreti for the patch. - Remove double quote in type definition en set type name in lower case when PRESERVE_CASE is disabled. - Add PREFIX_SUB_PARTITION to enable/disable sub-partitioning table prefixing in case of the partition names are a part of the sub- partition names. - Fix epoch replacement case in CREATE TABLE statements. - Apply epoch replacement to default value in table declaration. - Add special replacement for case of epoch syntax in Oracle: (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 is replaced by the PostgreSQL equivalent: (extract(epoch from now())) Thanks to rejo-oommen for the feature request. - A few typos in --help sections. Thanks to Christophe Courtois for the report. - Fix export of primary key on partition table. Thanks to chmanu for the patch. - Fix malformed user defined type export. Thanks to Francesco Loreti for the report. 2018 09 27 - v19.1 This release fix several issues reported during the last month and add support to PostgreSQL 11 HASH partitioning. It also adds some new features and configuration directives: * Add export of default partition and default sub partition. * Add export of HASH partition type. * Add support of stored procedure object. * Add replacement of NLSORT in indexes or queries. For example: CREATE INDEX test_idx ON emp (NLSSORT(emp_name, 'NLS_SORT=GERMAN')); is translated into CREATE INDEX test_idx ON emp ((emp_name collate "german")); The collation still need to be adapted, here probably "de_DE". NLSSORT() in ORDER BY clause are also translated. * Prevent duplicate index with primary key on partition to be exported. * PostgreSQL native partitioning does not allow direct import of data into already attached partitions. We now force direct import into main table but we keep Oracle export of data from individual This release also adds two new command line options: --oracle_speed: use to know at which speed Oracle is able to send data. No data will be processed or written --ora2pg_speed: use to know at which speed Ora2Pg is able to send transformed data. Nothing will be written Use it for debugging purpose. They are useful to see Oracle speed to send data and at what speed Ora2Pg is processing the data without reaching disk or direct import into PostgreSQL. Two new configuration directive has been added: * PG_SUPPORTS_PROCEDURE : PostgreSQL v11 adds support to stored procedure objects. Disabled by default. - PARALLEL_MIN_ROWS: set the minimum number of tuples in a table before calling Oracle's parallel mode during data export. Default to 100000 rows. Note that PG_SUPPORTS_PARTITION and PG_SUPPORTS_IDENTITY are now enabled by default to use PostgreSQL declarative partionning and identity column instead of serial data type. Here is the full list of changes and acknowledgements: - Fix automatic quoting of table or partition name starting with a number. Thanks to Barzaqh for the report. - Add information about custom directory installation. Thanks to joguess for the report. - Update list of action in documentation. - Fix export of spatial geometries. Thanks to burak yurdakul for the report. - Fix translation of default value in CREATE TABLE DDL when using a function. Thanks to Denis Oleynikov for the report. - Prevent moving index on partition during tablespace export. Thanks to Maxim Zakharov for the report. - Fix upper case of partition name in triggers. - Enforce KEEP_PKEY_NAMES when USE_TABLESPACE is enabled. Thanks to Maxim Zakharov for the patch. - Fix parsing of Oracle user login in dblink input from a file. - Fix multiple duplication of range clause in partition export. - Add bench of total time and rows to migrate data from Oracle in debug mode with speed average. - Fix sub partition prefix name. - Fix unset oracle username when exporting DBLINK from database. Thanks to Denis Oleynikov for the report. - Remove NO VALID to foreign keys on partitioned table. Thanks to Denis Oleynikov for the report. - Fix crash of Ora2Pg on regexp with dynamic pattern base on package code. Thank to Alain Debie and MikeCaliffCBORD for the report. - PostgreSQL native partitioning does not allow direct import of data into already attached partitions. When PG_SUPPORTS_PARTITION is enable we now force direct import into main single table but we keep Oracle export of data from individual partition. Previous behavior was to use main table from both side. Thanks to Denis Oleynikov for the report. - Add the PARALLEL_MIN_ROWS configuration directive to prevent Oracle's parallel mode to be activated during data export if the table have less than a certain amount of rows. Default is 100000 rows. This prevent unnecessary fork of Oracle process. Thanks to Denis Oleynikov for the feature request. - Fix composite partition MODULUS value. Thanks to Denis Oleynikov for the report. - Fix count of partitions that was not including subpartition count. - Force PostgreSQL user in FDW user mapping to be PG_USER when it is defined. - Sometimes Oracle indexes can be defined as follow: CREATE INDEX idx_err_status_id ON err_status (status_id, 1); which generate errors on PostgreSQL. Remove column names composed of digit only from the translation. Thanks to Denis Oleynikov for the report. - Move Oracle indexes or PK defined on partitioned tables to each partition as PostgreSQL do not support UNIQUE, PRIMARY KEY, EXCLUDE, or FOREIGN KEY constraints on partitioned tables. Definition are created in file PARTITION_INDEXES_output.sql generated with the PARTITION export type. Thanks to Denis Oleynikov for the feature request. - Fix parallel data load from Oracle partitioned tables by using a unique alias. Thanks to Denis Oleynikov for the report. - Fix export of composite partitioned (range/hash) table when PG_SUPPORTS_PARTITION is disabled. Thanks to Denis Oleynikov for the report. - Remove composite sub partition from the list of partition, this return a wrong partition count. - Fix MODULUS value in hash sub partitioning. - Index and table partitions could be on separate tablespaces. Thanks to Maxim Zakharov for the patch. - Fix case where procedure object name is wrongly double quoted. Thanks to danghb for the report. - Fix parser to support comment between procedure|function name and IS|AS keyword. Thanks to danghb for the report. - Remove dependency to List::Util for the min() function. 2018 08 18 - v19.0 This major release fix several issues reported by users during last year. It also adds several new features and configuration directives. New features: - Add export of Oracle HASH partitioning when PG_SUPPORTS_PARTITION is enabled. This is a PostgreSQL 11 feature. - Add SUBTYPE translation into DOMAIN with TYPE and PACKAGE export. - Add automatic translation of KEEP (DENSE_RANK FIRST|LAST ORDER BY ...) OVER (PARTITION BY ...) into FIRST|LAST_VALUE(...) OVER (PARTITION BY ... ORDER BY ...). - Add PCTFREE to FILLFACTOR conversion when PCTFREE is upper than the default value: 10. - Replace DELETE clause not followed with FROM (optional in Oracle). - Remove Oracle extra clauses in TRUNCATE command. - Allow use of NUMBER(*) in DATA_TYPE directive to convert all NUMBER(*) into the given type whatever is the length. Ex: DATA_TYPE NUMBER(*):bigint. - Add a PARALLEL hint to all Oracle queries used to migrate data. - Add export of Identity Columns from Oracle Database 12c. - Add translation of UROWID datatype and information in documentation about why default corresponding type OID will fail at data import. - Remove unwanted and unused keywords from CREATE TABLE statements: PARALLEL and COMPRESS. - Remove TEMPORARY in DROP statements. - Improve speed of escape_copy() function used for data export. - Add translation of Oracle functions NUMTOYMINTERVAL() and NUMTODSINTERVAL(). - Add counting of jobs defined in Oracle scheduler in the migration assessment feature. - Add CSMIG in the list of Oracle default system schema - Fully rewrite data export for table with nested user defined types DBD::Oracle fetchall_arrayref() is not able to associate complex custom types to the returned arrays, changed this call to use fetchrow_array() also used to export BLOB. - QUERY export will now output translated queries as well as untranslated ones. This break backward compatibility, previously only translated query was dumped. - Auto detect UTF-8 input files to automatically use utf8 encoding. - Support translation of MySQL global variables. - Add translation of preprocessor in Oracle external table into program in foreign table definition. Allow translation of external table from file. - Add translation to NVL2() Oracle function. - Translate CONVERT() MySQL function. - Translate some form of GROUP_CONCAT() that was not translated. - Remove call to CHARSET in cast() function, replace it by COLLATE every where else. This must cover most of the cases but some specials use might not, so please reports any issue with this behavior. - Add -c | --config command line option to ora2pg_scanner to set custom configuration file to be used instead of ora2pg default: /etc/ora2pg/ora2pg.conf - Improve CONNECT BY and OUTER JOIN translation. - And lot of MySQL to PostgreSQL improvements. Several new configuration directives have been added: - Add DEFAULT_PARALLELISM_DEGREE to control PARALLEL hint use when exporting data from Oracle. Default is disabled. - Make documentation about KEEP_PKEY_NAMES more explicit about kind of constraints affected by this directive. - Add PG_SUPPORTS_IDENTITY configuration directive to enable export of Oracle identity columns into PostgreSQL 10 feature. If PG_SUPPORTS_IDENTITY is disabled and there is IDENTITY column in the Oracle table, they are exported as serial or bigserial columns. When it is enabled they are exported as IDENTITY columns like: CREATE TABLE identity_test_tab ( id bigint GENERATED ALWAYS AS IDENTITY, description varchar(30) ) ; If there is non default sequence option set in Oracle, they will be appended after the IDENTITY keyword. Additionally in both cases Ora2Pg will create a file AUTOINCREMENT_output.sql with a function to update the associated sequences with the restart value set to "SELECT max(colname)+1 FROM tablename". Of course this file must be imported after data import otherwise sequence will be kept to start value. - Add DISABLE_UNLOGGED configuration directive. By default Ora2Pg export Oracle tables with the NOLOGGING attribute into UNLOGGED tables. You may want to fully disable this feature because you will lost all data from unlogged table in case of PostgreSQL crash. Set it to 1 to export all tables as normal table. When creating a new migration project using --init_project, this directive is activated by default. This is not the case in the default configuration file for backward compatibility. - Add FORCE_SECURITY_INVOKER configuration directive. Ora2Pg use the function's security privileges set in Oracle and it is often defined as SECURITY DEFINER. To override those security privileges for all functions and use SECURITY DEFINER instead, enable this directive. - Add AUTONOMOUS_TRANSACTION in configuration to enable translation of autonomous transactions into a wrapper function using dblink or pg_background extension. If you don't want to use this feature and just want to export the function as a normal one without the pragma call, disable this directive. - Add documentation about COMMENT_SAVEPOINT configuration directive. - Major rewrite in PACKAGE parser to better support global variables detection. Global variable that have no default values are now always initialized to empty string in file global_variables.conf so that we see that they exists. This might not change the global behavior. I especially want to thank Pavel Stehule and Eric Delanoe who spent lot of time this year to help me to improve the PL/SQL to plpgsql translation and also Krasiyan Andreev who help a lot to finalize the MySQL to PostgreSQL migration features. Here is a complete list of changes and acknowledgments: - Fix translation of "varname cursor%ROWTYPE;". Thanks to Philippe Beaudoin for the report. - Fix return of autonomous transaction dblink call when function has OUT parameter. Thanks to Pavel Stehule for the report. - Add Oracle to PostgreSQL translation of windows functions KEEP (DENSE_RANK FIRST|LAST ORDER BY ...) OVER (PARTITION BY ...) Thanks to Swapnil bhoot929 for the feature request. - Fix "ORA-03113: end-of-file on communication channel" that what generated by a too long query send to Oracle. The size of queries sent to Oracle to retrieve object information depend of the ALLOW and EXCLUDE directives. If you have lot of objects to filter you can experience this kind of non explicit error. Now Ora2pg use bind parameter to pass the filters values to reduce the size of the prepared query. Thanks to Stephane Tachoire for the report. - Add SUBTYPE translation into DOMAIN with TYPE and PACKAGE export. Thanks to Francesco Loreti for the feature request. - Fix PLS_INTEGER replacement. - Remove precision for RAW|BLOB as type modifier is not allowed for type "bytea". - Fix call of schema.pckg.function() in indexes with a replacement with pckg.function(). Thanks to w0pr for the report. - Fix translation of UPDATE trigger based on columns: "BEFORE UPDATE OF col1,col2 ON table". Thanks to Eric Delanoe for the report. - Remove single / from input file that was causing a double END in some case. Thanks to Philippe Beaudoin for the report. - Limit translation of PCTFREE into FILLFACTOR when PCTFREE is upper than the Oracle default value: 10. With PostgreSQL 100 (complete packing) is the default. - Add PCTFREE to FILLFACTOR conversion. Thanks to Maxim Zakharov for the patch. - Remove TRUNCATE extra clauses. Thanks to e7e6 for the patch. - Fix type conversion when extra \n added after ;. Thanks to Maxim Zakharov for the patch. - Fix DELETE clause not followed with FROM (optional in Oracle). Thanks to Philippe Beaudoin for the patch. - Limit call to ALL_TAB_IDENTITY_COLS to version 12+. Thanks to Andy Garfield for the report. - Fix comment parsing. Thanks to Philippe Beaudoin for the report. - Allow use of NUMBER(*) in DATA_TYPE directive to convert all NUMBER(*) into the given type whatever is the length. Thanks to lingeshpes for the feature request. - Fix bug in function-based index export. Thanks to apol1234 for the report. - Add PARALLEL hint to all data export queries. Thanks to jacks33 for the report. - Make documentation about KEEP_PKEY_NAMES more explicit about kind of constraints affected by this directive. - Fix export of identity columns by enclosing options between parenthesis and replacing CACHE 0 by CACHE 1. Thanks to swmcguffin devtech for the report. - Add parsing of identity columns from file. - Fix unwanted replacement of IF () in MySQL code. Thanks to Krasiyan Andreev for the report. - Fix to_char() translation, thanks to Eric Delanoe for the report. - Fix untranslated PERFORM into exception. Thanks to Pavel Stehule for the report. - Add _get_entities() function to MySQL export. It returns nothing, AUTO_INCREMENT column are translated with corresponding types, smallserial/serial/bigserial. - Fix look at encrypted column on Oracle prior to 10. Thanks to Stephane Tachoires for the patch. - Add export of Identity Columns from Oracle Database 12c. Thanks to swmcguffin-devtech for the feature request. - Prevent Ora2Pg to scan ALL_SCHEDULER_JOBS for version prior to 10 Thanks to Stephane Tachoires for the patch. - Fix pull request #648 to log date only when debug is enabled and use POSIX strftime instead of custom gettime function. - Add system time to debug log info. Thanks to danghb for the patch. - Fix parsing of trigger from file and exception. - Fix very slow export of mysql tablespace when number of table is large. Thanks to yafeishi for the report. - Fix translation of CAST( AS unsigned). Thanks to Krasiyan Andreev. - Fix MySQL character length to use character_maximum_length instead of equal character_octet_length. Thanks to yafeishi for the report. - Fix custom replacement of MySQL data type. Thanks to Krasiyan Andreev for the report. - Fix replacement of call to open cursor with empty parenthesis. Thanks to Philippe Beaudoin for the report. - Fix MySQL data type conversion in function declaration. Thanks to Krasiyan Andreev for the report. - Fix error with -INFINITY as default value for date or timestamp columns. - Fix procedure call rewrite with unwanted comma on begin of parameter list. Thanks to Pavel Stehule for the report. - Fix handling of foreign keys when exporting data and DROP_FKEYS is enabled and ALLOW/EXCLUDE directive is set. Now Ora2Pg will first drop all foreign keys of a table in the export list and all foreign keys of other tables pointing to the table. After data import, it will recreate all of these foreign keys. Thanks to Eric Delanoe for the report. - Fix broken transformation of procedure call with default parameter Thanks to Pavel Stehule for the report. - Translate call to TIMESTAMP in partition range values into a cast. Thanks to markiech for the report. - Fix CONNECT BY translation when the query contain an UNION. Thanks to mohammed-a-wadod for the report. - Fix CONNECT BY with PRIOR on the right side of the predicat. - Fix outer join translation when the (+) was in a function, ex: WHERE UPPER(trim(VW.FRIDAY))= UPPER(trim(FRIDAY.NAME(+))). - Order outer join pending tables in from clause. - Order by object name comments and indexes export. - Fix outer join translation when the table is not in the from clause. Thanks to Cyrille Lintz for the report. - Try to fix potential Oracle schema prefixing PostgreSQL schema name in CREATE SCHEMA. Thanks to Cyrille Lintz for the report. - Fix error in TRIM() translation. Thanks to Cyrille Lintz for the report. - Add translation of UROWID datatype and information in documentation about why default corresponding type OID will fail at data import. Thanks to Cyrille Lintz for the report. - Fix bug in exporting boolean default values in column definition. - Fix bug in column parsing in CREATE TABLE. - Adapt default value for data type changed to boolean. - Fix bad handling of -D (data_type) option. - Change behavior in the attempt to set MySQL global variable type. Now variable type will be timestamp if the variable name contains datetime, time if the name contains only time and date for date. Thanks to Krasiyan Andreev for the report. - Fix function replacement in MySQL declare section. Thanks to Krasiyan Andreev fr the report. - Apply REPLACE_ZERO_DATE to default value in table declaration. Thanks to Krasiyan Andreev for the report. - Add support to embedded comment in table DDL. - Fix replacement of data type for MySQL code. Thanks to Krasiyan Andreev for the report. - Fix MySQL type replacement in function. Thanks to Krasiyan Andreev for the report. - Improve speed of escape_copy() function used for data export. Thanks to pgnickb for the profiling. - Add translation of Oracle functions NUMTOYMINTERVAL() and NUMTODSINTERVAL(). Thanks to Pavel Stehule for the report. - Counting jobs defined in Oracle scheduler. Thanks to slfbovey for the patch. - Fix several issue in create table DDL parser: - remove double quote of object name when a list of column is entered - split of table definition to extract column and constraint parts is now more efficient - remove dot in auto generated constraint name when a schema is given in table name - fix default values with space that was breaking the parser - Remove use of bignum perl module that reports error on some installation. Thanks to Cyrille Lintz for the report. - Fix a typo preventing perldoc to complete. Thanks to slfbovey for the patch. - Fully rewrite data export for table with nested user defined types DBD::Oracle fetchall_arrayref() is not able to associate complex custom types to the returned arrays, changed this call to use fetchrow_array() also used to export BLOB. Thanks to lupynos for the report. - Fix renaming of temporary files during partitions data export. - Fix Oracle use of empty string as default value for integers. Oracle allow such declaration: SOP NUMBER(5) DEFAULT '' which PostgreSQL does not support. Ora2Pg now detect this syntax and replace empty string with NULL. Thanks to ricdba for the report. - Add detection of Oracle version before setting datetime format, needed for Oracle 8i compatibility. - Export of tables from Oracle database are now ordered by name by default. Thanks to Markus Roth for the report. - Fix an other case of missing translation of UNSIGNED into bigint. Thanks to Krasiyan Andreev for the report. - Force replacement of double quote into single quote for MySQL view and function code. - Fix case when SET @varname := ... is used multiple time in the same function. Thanks to Krasiyan Andreev for the report. - Fix case where SET @varname := ... was not translated. Thanks to Krasiyan Andreev for the report. - Adjust the regex pattern of last patch. - Fix unwanted newline after hint replacement that could break comments. Thanks to Pavel Stehule for the report. - Fix if() replacement in query. Thanks to Krasiyan Andreev for the report. - Remove extra parenthesis in some form of JOIN. Thanks to Krasiyan Andreev for the report. - Fix untranslated call to UNSIGNED, now translated as bigint. - Thanks to Krasiyan Andreev for the report. - Fix translation of double(p,s) into decimal(p,s). - Remove use of SET when an assignment is done through a SELECT statement. Thanks to Krasiyan Andreev for the report. - Fix non-quoted reserved keywords in INSERT / COPY statements when exporting data. Thanks to Pavel Stehule for the report. - Fix partition data export to file, temporary files for partition output was not renamed at export end then data was not loaded. - Fix double operator := during function with out param rewrite. - Fix commit f1166e5 to apply changes when FILE_PER_FUNCTION is disable or when an input file is given. - Fix translation of LOCATE(). Thanks to Krasiyan Andreev for the report. - Fix case where MySQL GROUP_CONCAT() function was not translated. Thanks to Krasiyan Andreev for the report. - Fix :new and :old translation in triggers. - Fully rewrite function call qualification process, the second pass now is only use to requalify call to pkg.fct into pkg_ftc when PACKAGE_AS_SCHEMA is disable. The replacement of all function calls using double quote when a non supported character is used or when PRESERVE_CASE is enabled has been completely removed as this takes too much time to process for just very few case. So by default now Ora2Pg will not go through the second pass. This can change in the future especially if this is more performant to process PERFORM replacement. Thanks a lot to Eric Delanoe for his help on this part. - Exclude function and procedure not from package to be used in requalify call. Thanks to Eric Delanoe for the report. - Fix function name qualification in multiprocess mode. - Fix unqualified function call due to unclose file handle. - Prevent try to requalify function call if the function is not found in the file content. - Remove ALGORITHM=.*, DEFINER=.* and SQL SECURITY DEFINER from MySQL DDL code. - An other missing change to previous commit on qualifying function call. - Limit function requalification to export type: VIEW, TRIGGER, QUERY, FUNCTION, PROCEDURE and PACKAGE. - Auto detect UTF-8 input files to automatically use utf8 encoding. - Remove all SHOW ERRORS and other call to SHOW in Oracle package source as they was badly interpreted as global variable. - Fix MySQL CREATE TABLE ... SELECT statement. - Fix pending translation issue on some DATE_FORMAT() case. Thanks to Krasiyan Andreev for the report. - Fix translation of IN (..) in MySQL view. Thanks to Krasiyan Andreev for the report. - Fix MySQL date format with digit. - Fix DATE_FORMAT, WHILE and IFNULL translation issues. - Fix not translated MySQL IF() function. - Fix other MySQL translation issues for @variable. Thanks to Krasiyan Andreev for the report. - Fix issue in MySQL IF translation with IN clause. Thanks to Krasiyan Andreev for the report. - Clarify comment about XML_PRETTY directive. Thanks to TWAC for the report. - Fix remaining MySQL translation issues for @variable reported in issue #590. - Fix no translated := in SET statement. - Fix output order of translated function. - Fix non printable character or special characters that make file encoding to ISO-8859 instead of utf8. Thanks to twac for the report. - Prevent MySQL global variable to be declared twice. Thanks to Krasiyan Andreev for the report. - Support translation of MySQL global variables. Session variable @@varname are translated to PostgreSQL GUC variable and global variable @varname are translated to local variable defined in a DECLARE section. Ora2Pg tries to gather the data type by using integer by default, varchar if there is a constant string ('...') in the value and a timestamp if the variable name have the keyword date or time inside. Thanks to Krasiyan Andreev for the feature request. - Fix DATE_ADD() translation. - Add translation of preprocessor in Oracle external table into program in foreign table definition. Thanks to Thomas Reiss for the report. Allow translation of external table from file. - Fix case where IF EXISTS might not be append when it is not supported by PG. - Translate CONVERT() MySQL function. Thanks to Krasiyan Andreev for the report. - Translate some form of GROUP_CONCAT() that was not translated. Thanks to Krasiyan Andreev for the report. - Apply same principe with COMMIT in MySQL function code than in Oracle code. It is kept untouched to be able to detect a possible change of code logic. It can be automatically commented if COMMENT_COMMIT_ROLLBACK is enabled. Also I have kept the START TRANSACTION call but it is automatically commented. - Add mysql_enable_utf8 => 1 to MySQL connection to avoid issues with encoding. Thanks to Krasiyan Andreev for the report. - Prevent removing of comment on MySQL function and add a "COMMENT ON FUNCTION" statement at end of the function declaration. Thanks to Krasiyan Andreev for the report. - Fix translation of types in MySQL function parameter. Thanks to Krasiyan Andreev for the report. - Remove START TRANSACTION from MySQL function code. Thanks to Krasiyan Andreev for the report. - Fix previous patch, we do not need to look forward for function or procedure definition in VIEW export and there is no package with MySQL. Thanks to Krasiyan Andreev for the report. - Fix call to useless function for MySQL function. - Add rewrite of MySQL function call in function or procedure code translation and some other translation related to MySQL code. - Fix ora2pg_scanner when exporting schema with $ in its name. Thanks to Aurelien Robin for the report. - Disable number of microsecond digit for Oracle version 9. Thanks to Aurelien Robin for the report. - Do not look at encrypted column for DB version < 10. Thanks to Aurelien Robin for the report. - Fix MySQL call to charset in cast function. MySQL charset "utf8" is also set to COLLATE "C.UTF-8". Thanks to Krasiyan Andreev for the report. - Fix two bug in CONNECT BY and OUTER JOIN translation. - Forgot to handle exception to standard call to IF in MySQL IF() translation. Thanks to Krasiyan Andreev for the report. - Forgot to apply previous changes to procedure. - Fix IF() MySQL replacement when nested and when containing an IN (...) clause. Thanks to Krasiyan Andreev for the report. - Fix double BEGIN on MySQL function export. Thanks to Krasiyan Andreev for the report. - Fix enum check constraint name when PRESERVE_CASE is enabled. - Fix case where object with LINESTRING and CIRCULARSTRING was exported as MULTILINESTRING instead of MULTICURVE. - Fix export of MULTICURVE with COMPOUNDCURVE. Thanks to Petr Silhak for the report. - Fix several issue in MySQL table DDL export. Thanks to Krasiyan Andreev for the report. - Fix MySQL auto_increment data type translation and columns export order. - Fix translation of MySQL function CURRENT_TIMESTAMP(). Thanks to Krasiyan Andreev for the report. - Fix export of MySQL alter sequence name when exporting auto increment column. Thanks to Krasiyan Andreev for the report. - Replace IF() call with CASE ... END in VIEW and QUERY export for MySQL. Thanks to Krasiyan Andreev for the feature request. - Replace backquote with double quote on mysql statements when read from file. - Fix bug in REGEXP_SUBSTR replacement. - Prevent replacement with same function name from an other package. Thanks to Eric Delanoe for the report. - Apply same STRICT rule for SELECT INTO to EXECUTE INTO. Thanks to Pavel Stehule for the report. - Fix extra parenthesis removing when a OR clause is present. Thanks to Pavel Stehule for the report. - Keep autonomous pragma commented when conversion is deactivated to be able to identify functions using this pragma. - Fix bug in replacement of package function in string constant. - Fix malformed replacement of array element calls. Thanks to Eric Delanoe for the report. - Fix unwanted replacement of TO_NUMBER function. Thanks to Torquem for the report. - Add an example of DSN for MySQL in ORACLE_DSN documentation. Thanks to François Honore for the report. - Fix typo in default dblink connection string. Thanks to Pavel Stehule for the report. - Add information about Oracle Instant Client installation. Thanks to Jan Birk for the report. - Replace Oracle array syntax arr(i).x into arr[i].x into PL/SQL code. Thanks to Eric Delanoe for the report. - Use a more generic connection string for DBLINK. It will use unix socket by default to connect and the password must be set in .pgpass. This will result in the following connection string: format('port=%s dbname=%s user=%', current_setting('port'), current_database(), current_user) If you want to redefine this connection string use DBLINK_CONN configuration directive. Thanks to Pavel Stehule for the feature request. - Fix missing RETURN NEW in some trigger translation. Thanks to Pavel Stehule for the report. - Fix a missing but non mandatory semi-comma. - Keep PKs/unique constraints which are deferrable in Oracle also deferrable in PostgreSQL. Thank to Sverre Boschman for the patch. - Fix parsing and translation of CONNECT BY. Thanks to bhoot929 for the report. - Fix FDW export when exporting all schema. Thanks to Laurenz Albe for the report. - Add a note about multiple value in export type that can not include COPY or INSERT together with others export type. - Fix duplicate condition. Thanks to Eric Delanoe for the report. - Fix unwanted translation into PERFORM after INTERSECT. - Comment savepoint in code. Thanks to Pavel Stehule for the patch. - Fix "ROLLBACK TO" that was not commented. Thanks to Pavel Stehule for the report. - Fix restore of constant string when additional string constant regex are defined in configuration file. - Fix translation of nextval with sequence name prefixed with their schema. - Cast call to TO_DATE(LOCALTIMESTAMP,...) translated into TO_DATE(LOCALTIMESTAMP::text,...). Thanks to Keshav kumbham for the report. - Remove double quote added automatically by Oracle on view definition when PRESERVE_CASE is not enable. Thanks to JeeIPI for the report. - Fix translation of FROM_TZ with a call to function as first parameter. Thanks to TrungPhan for the report. - Fix package export when FILE_PER_FUNCTION is set. Thanks to Julien Rouhaud for the report. - Add translation of REGEXP_SUBSTR() with the following rules: Translation of REGEX_SUBSTR( string, pattern, [pos], [nth]) converted into SELECT array_to_string(a, '') FROM regexp_matches(substr(string, pos), pattern, 'g') AS foo(a) LIMIT 1 OFFSET (nth - 1); Optional fifth parameter of match_parameter is appended to 'g' when present. Thanks to bhoot929 for the feature request. - Add count of REGEX_SUBSTR to migration assessment cost. - Add translation support of FROM_TZ() Oracle function. Thanks to trPhan for the feature request. - Forces ora2pg to output a message when a custom exception code has less than 5 digit. - Fix errcode when Oracle custom exception number have less than five digit. Thanks to Pavel Stehule for the report. - Fix case where custom errcode are not converted. Thanks to Pavel Stehule for the report. - Fix print of single semicolon with empty line in index export. - Fix problem with TO_TIMESTAMP_TZ conversion. Thanks to Keshav- kumbham for the report. - Fix unwanted double quote in index column with DESC sorting. Thanks to JeeIPI for the report. - Fix non detection case of tables in from clause for outer join translation. Thanks to Keshav for the report. - Fix unwanted replacement of = NULL into IS NULL in update statement. Thanks to Pavel Stehule for the report. - Force schema name used in TEST action to lowercase. Thanks to venkatabn for the report. - Fix export of spatial geometries with CURVEPOLYGON + COMPOUNDCURVE Thanks to kabog for the report. 2017 09 01 - v18.2 This release fix several issues reported during the last six months. It also adds several new features and configuration directives: - Add translation of SUBSTRB into substr. - Allow use of array in MODIFY_TYPE to export Oracle user defined type that are just array of some data type. For example: CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); can be directly translated into text[] or varchar[]. In this case use the directive as follow: MODIFY_TYPE CLUB:MEMBERS:text[] Ora2Pg will take care to transform all data of this column into the correct format. Only arrays of characters and numerics types are supported. - Add translation of Oracle function LISTAGG() into string_agg(). - Add TEST_VIEW action to perform a simple count of rows returned by views on both database. - Translate SQL%ROWCOUNT into GET DIAGNOSTICS rowcount = ROW_COUNT and add translation of SQL%FOUND. - Add translation of column in trigger event test with IS DISTINCT, for example: IF updating('ID') THEN ... will be translated into: IF TG_OP = 'UPDATE' AND NEW.'ID' IS DISTINCT FROM OLD.'ID' then... - Replace UTL_MATH.EDIT_DISTANCE function by fuzzymatch levenshtein. - Allow use of MODIFY_STRUCT with TABLE export. Table creation DDL will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. - Add export of partition and subpartition using PostgreSQL native partitioning. - Auto detect encrypted columns and report them into the assessment. SHOW_COLUMN will also mark columns as encrypted. - Add information to global temporary tables in migration assessment. - Add experimental DATADIFF functionality. - Allow use of multiprocess with -j option or JOBS to FUNCTION and PROCEDURE export. Useful if you have thousands of these objects. - Force RAW(N) type with default value set to sys_guid() as UUID on PostgreSQL. - Replace function with out parameter using select into. For example a call to: get_item_attr( attr_name, p_value ); where p_value is an INOUT parameter, will be rewritten as p_value := get_item_attr( attr_name, p_value ); If there is multiple OUT parameters, Ora2Pg will use syntax: SELECT get_item_attr( attr_name, p_value ) INTO (attr_name, p_value); - Add translation of CONNECT BY using PostgreSQL CTE equivalent. This translation also include a replacement of LEVEL and SYS_CONNECT_BY_PATH native Oracle features. On complex queries there could still be manual editing but all the main work is done. - Add support to user defined exception, errcode affected to each custom exception start from 50001. - Translate call to to_char() with a single parameter into a cast to varchar. Can be disabled using USE_ORAFCE directive. - Improve ora2pg_scanner to automatically generates migration assessment reports for all schema on an Oracle instance. Before the schema name to audit was mandatory, now, when the schema is not set Ora2Pg will scan all schema. The connexion user need to have DBA privilege. Ora2Pg will also add the hostname and SID as prefix in the filename of the report. This last change forbids ora2pg_scanner to overwrite a report if the same schema name is found in several databases. Several new configuration directives have been added: - Add USE_ORAFCE configuration directive that can be enabled if you want to use functions defined in the Orafce library and prevent Ora2Pg to translate call to these functions. The Orafce library can be found here: https://github.com/orafce/orafce By default Ora2pg rewrite add_month(), add_year(), date_trunc() and to_char() functions, but you may prefer to use the Orafce functions that do not need any code transformation. Directive DATE_FUNCTION_REWRITE has been removed as it was also used to disable replacement of add_month(), add_year() and date_trunc() when Orafce is used, useless now. - Add FILE_PER_FKEYS configuration directive to allow foreign key declaration to be saved in a separate file during schema export. By default foreign keys are exported into the main output file or in the CONSTRAINT_output.sql file. If enabled foreign keys will be exported into a file named FKEYS_output.sql - Add new COMMENT_COMMIT_ROLLBACK configuration directive. Call to COMMIT/ROLLBACK in PL/SQL code are kept untouched by Ora2Pg to force the user to review the logic of the function. Once it is fixed in Oracle source code or you want to comment this calls enable the directive. - Add CREATE_OR_REPLACE configuration directive. By default Ora2Pg use CREATE OR REPLACE in function DDL, if you need not to override existing functions disable this configuration directive, DDL will not include OR REPLACE. - Add FUNCTION_CHECK configuration directive. Disable this directive if you want to disable check_function_bodies. SET check_function_bodies = false; It disables validation of the function body string during CREATE FUNCTION. Default is to use de postgresql.conf setting that enable it by default. - Add PG_SUPPORTS_PARTITION directive, disabled by default. PostgreSQL version prior to 10.0 do not have native partitioning. Enable this directive if you want to use PostgreSQL declarative partitioning instead of the old style check constraint and trigger. - Add PG_SUPPORTS_SUBSTR configuration directive to replace substr() call with substring() on old PostgreSQL versions or some fork like Redshift. - Add PG_INITIAL_COMMAND to send some statements at session startup. This directive is the equivalent used for Oracle connection, ORA_INITIAL_COMMAND. Both can now be used multiple time now. - Add DBLINK_CONN configuration directive. By default if you have an autonomous transaction translated using dblink extension the connection is defined using the values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override the connection string use this directive to set the connection in the autonomous transaction wrapper function. For example: DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass - Add STRING_CONSTANT_REGEXP configuration directive. Ora2Pg replace all string constant during the pl/sql to plpgsql translation, string constant are all text include between single quote. If you have some string placeholder used in dynamic call to queries you can set a list of regexp to be temporary replaced to not break the parser. For example: STRING_CONSTANT_REGEXP The list of regexp must use the semi colon as separator. - Add FUNCTION_STABLE configuration directive. By default Oracle functions are marked as STABLE as they can not modify data unless when used in PL/SQL with variable assignment or as conditional expression. You can force Ora2Pg to create these function as VOLATILE by disabling this configuration directive. - Add new TO_NUMBER_CONVERSION configuration directive to control TO_NUMBER translation behavior. By default Oracle call to function TO_NUMBER will be translated as a cast into numeric. For example, TO_NUMBER('10.1234') is converted into PostgreSQL call: to_number('10.1234')::numeric. If you want you can cast the call to integer or bigint by changing the value of the configuration directive. If you need better control of the format, just set it as value, for example: TO_NUMBER_CONVERSION 99999999999999999999D9999999999 will convert the code above as: TO_NUMBER('10.1234', '99999999999999999999D9999999999') Any value of the directive that it is not numeric, integer or bigint will be taken as a mask format. If set to none, then no conversion will be done. - Add LOOK_FORWARD_FUNCTION configuration directive which takes a list of schema to get functions/procedures meta information that are used in the current schema export. When replacing call to function with OUT or INOUT parameters, if a function is declared in an other package then the function call rewriting can not be done because Ora2Pg only knows about functions declared in the current schema. By setting a comma separated list of schema as value of the directive, Ora2Pg will look forward in these packages for all functions, procedures and packages declaration before proceeding to current schema export. - Add PG_SUPPORTS_NAMED_OPERATOR to control the replacement of the PL/SQL operator used in named parameter => with the PostgreSQL proprietary operator := Disable this directive if you are using PG < 9.5 - Add a warning when Ora2Pg reorder the parameters of a function following the PostgreSQL rule that all input parameters following a parameter with a default value must have default values as well. In this case, Ora2Pg extracts all parameters with default values and put them at end of the parameter list. This is to warn you that a manual rewrite is required on calls to this function. New command line options have been added: - Add -N | --pg_schema command line option to be able to override the PG_SCHEMA configuration directive. When this option is set at command line, EXPORT_SCHEMA is automatically activated. - Add --no_header option with equivalent NO_HEADER configuration directive to output the Ora2Pg header but just the translated code. There is also some behavior changes from previous release: - Remove SysTimestamp() from the list of not translated function, it is replaced with CURRENT_TIMESTAMP for a long time now. - Change migration assessment cost to 84 units (1 day) for type TABLE, INDEX and SYNONYM and to 168 units (2 days) for TABLE PARTITION and GLOBAL TEMPORARY TABLE, this is more realistic. - Set minimum assessment unit to 1 when an object exists. Improve PL/SQL code translation speed. - Change behavior of COMPILE_SCHEMA directive used to force Oracle to compile schema before exporting code. When this directive is enabled and SCHEMA is set to a specific schema name, only invalid objects in this schema will be recompiled. When SCHEMA is not set then all schema will be recompiled. To force recompile invalid object in a specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. This will ask to Oracle to validate the PL/SQL that could have been invalidate after a export/import for example. The 'VALID' or 'INVALID' status applies to functions, procedures, packages and user defined types. - Default transaction isolation level is now set to READ COMMITTED for all action excluding data export. - Oracle doesn't allow the use of lookahead expression but you may want to exclude some objects that match the ALLOW regexp you have defined. For example if you want to export all table starting with E but not those starting with EXP it is not possible to do that in a single expression. Now you can start a regular expression with the ! character to exclude all objects matching the regexp given just after. Our previous example can be written as follow: ALLOW E.* !EXP.* it will be translated into REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') in the object search expression. - Fix quoting of PG_SCHEMA with multiple schema in search path. The definition of the search path now follow the following behavior: * when PG_SCHEMA is define, always set search_path to its value. * when EXPORT_SCHEMA is enabled and SCHEMA is set, the search_path is set the name of the schema. - Remove forcing of export_schema when pg_schema is set at command line. This could change the behavior of some previous use of these variables and the resulting value of the search_path but it seems much better understandable. - Rewrite translation of raise_application_error to use RAISE EXCEPTION with a message and the SQLSTATE code. Oracle user defined code -20000 to -20999 are translated to PostgreSQL user define code from 45000 to 45999. Call to raise_application_error(mySQLCODE, myErrmsg); will be translated into RAISE EXCEPTION '%', myErrmsg USING ERRCODE = mySQLCODE; - Remove migration assessment cost for TG_OP and NOT_FOUND they might be fully covered now. Here is the complete list of changes: - Fix bad inversion of HAVING/GROUP BY clauses. Thanks to bhoot929 for the report. - Fix case of non translation of type in CAST() function. Thanks to Keshavkumbham for the report. - Fix spatial data export when using direct import into PostgreSQL and WKT or INTERNAL format. This can still be improved. Thanks to Valeria El-Samra for the report. - Improve translation of trunc() into date_trunc. Thanks to bhoot929 for the report. - Translate to_char() without format into a simple cast to varchar. Thanks to bhoot929 for the report. - Fix line comment which does not disable multi-line comment. Thanks to Pavel Stehule for the report. - Fix overridden of output file global_variables.conf with multiple packages. Thanks to Oliver Del Rosario for the report. - Fix export of data stored in a nested user defined type. Thanks to lupynos for the report. - Fix data export from Oracle user defined types, where output of ROW(...) does not distinguish numeric from string or other types that need to be formatted. Thanks to Petr Silhak for the report. - Fix broken replacement of package procedure name. Thanks to Pavel Stehule for the report. - Add FLOWS_010600 to the objects exclusion listr. - Improve view/trigger migration assessment accuracy. - Fix OUTER JOIN (+) translation, all join filters with constant was written into the WHERE clause by default. Write them into the JOIN clause. - Fix weight of the number of triggers and views in the report with a limit of 2 man-days, of course SQL difficulties are still add after this limit. - Fix alias added to condition which is not a sub query. Thanks to nitinmverma for the report. - Fix wrong translation of OUTER JOIN with subquery in FROM clause. Thanks to nitinmverma for the report. - Fix typo preventing exclusion of system SYNONYM. - Fix an other case of bad translation of END fct_name. Thanks to nitinmverma for the report. - Fix unwanted translation of REGEXP_SUBSTR() in REGEXP_SUBSTRING(). Thanks to nitinmverma for the report. - Fix broken translation of decode(). Thanks to nitinmverma for the report. - Fix error "Malformed UTF-8 character (unexpected continuation byte 0xbf, with no preceding start byte) in pattern match" by including an arbitrary non-byte character into the pattern. Thanks to Bob Sislow for the report. - Fix missing translation of trunc() with date_trunc(). Thanks to nitinmverma for the report. - Add migration assessment weight to concatenation. - Fix space between operator, ex: a < = 15 must be translated as a <= 15. Thanks to nitinmverma for the report. - Handle translation of named parameters in function calls. Thanks to Julien Rouhaud for the patch. - Fix missing renaming of _initial_command() method. - Fix right outer join translation by converting them to left outer join first. Thanks to Julien Rouhaud for the hint. - Fix TEST on default value count and functions belonging to others than public schema, especially functions of packages. - Fix default number of man-days in migration assessment. Thanks to Nate Fitzgerald for the report. - Add host information into the filename of the report to prevent some additional case of report overriding. Thanks to Aurelien Robin for the report. - Prevent ora2pg script to complain if no ora2pg.conf file is found when a DSN is passed at command line and that user connection is set in the environment variables. - Do not declare a function stable if there is update/insert/delete statement inside. - Improve ora2pg_scanner to generate report of each instance schema when the schema to audit is not set. Thanks to Thomas Reiss for the patch. - Fix parser failure with quote in comments. Thanks to Eric Delanoe for the report. - Fix case where NVL is not replaced by COALESCE. - Add parenthesis to simple package.function call without parameter. - Fix replacement of INSTR() with optional parameters. Thanks to Pavel Stehule for the report. - Translate SQL%ROWCOUNT to GET DIAGNOSTICS rowcount = ROW_COUNT. Thanks to Pavel Stehule for the report. - Add translation of SQL%FOUND. Thanks to Pavel Stehule. - Remove qualifier in create type, "CREATE TYPE fusion.mytable AS (fusion.mytable fusion.finalrecord[]);" becomes "CREATE TYPE fusion.mytable AS (mytable fusion.finalrecord[]);". Thanks to Julien Rouhaud for the report. - Fix extra comma in FROM clause of triggers in outer join translation. Thanks to Pavel Stehule fora the report. - Use record for out parameters replacement only where there is more than one out parameter. Thanks to Pavel Stehule for the patch. - Add date type to the inout type array. Thanks to Pavel Stehule for the report. - Remove possible last spaces in inout type detection. - Fix REGEXP_LIKE translation. - Fix count of default values during test action. - Fix removing of double quote over column name in view declaration. - Do not set default value if it is NULL, this is already the case. - Fix data export that was truncated to the last DATA_LIMIT lines. Thanks to Michael Vitale for the report. - Fix an other bug in rewriting call to function with OUT parameter. Thanks to Pavel Stehule for the report. - Fix autodetection of composite out parameters. - Merge typo on PLPGSQL - Fix typo to PLPGSQL keyword. Thanks to Vinayak Pokale. - Fix regexp failure in is_reserved_words() method. Thanks to Patrick Hajek for the patch. - Use only one declaration of ora2pg_r RECORD, it is reusable. - Fix transformation of procedure CALL with OUT parameter that can't works when procedure/function has minimally one OUT parameter is of composite type. Thanks to Pavel Stehule for the patch. - Second attempt to fix outer join translation in triggers. Thanks to Pavel Stehule for precious help. - Fix RAISE NOTICE replacement with double % placeholder. Thanks to Pavel Stehule for the report. - Fix call to function replacement for function registered with double quote. Thanks to Pavel Stehule for the report. - Change assessment score of TG_OP. - Fix replacement of outer join in triggers by adding pseudo tables NEW and OLD to the list of tables. Thanks to Pavel Stehule for the report. - Handle custom exception in declare section of triggers. - Fix FUNCTION_CHECK option, it will be set in all file header. Thanks to Pavel Stehule for the report. - Replace call to ALL_TABLES to USER_TABLES when USER_GRANTS is enabled. Thanks to Bob Sislow. - Removed PRAGMA EXCEPTION_INIT() from declare section. Thanks to Pavel Stehule for the report. - Fix constant string that was breaking the parser. Thanks to Pavel Stehule for the report. - Fix missing space between EXCEPTION and WHEN. Thanks to Pavel Stehule for the report. - Fix broken function header resulting in missing space before OUT keyword. Thanks to Pave Stehule for the report. - Fix invalid RAISE command. Thanks to Pavel Stehule for the report. - Add port information to ORACLE_DSN in documentation and config file. Thanks to Bob Sislow for the report. - Fix broken declaration in triggers related to FOR cycle control variable when there is no declare section. Thanks to Pavel Stehule for the report. - Fix broken declaration in triggers related to FOR cycle control variable. Thanks to Pavel Stehule for the report. - Fix unterminated C style comment in trigger. Thanks to Pavel Stehule for the report. - Fix bug in package+function precedence replacement. Thanks to Eric Delanoe for the report. - Fix unwanted and broken export of tables created with CREATE TABLE tablename OF objType. Thanks to threenotrump for the report. - Add explanation on using REPLACE_AS_BOOLEAN when REPLACE_TABLES or REPLACE_COLS is also used on the same object. Thanks to Brendan Le Ny for the report. - Fix unwanted data export of materialized view. Thanks to Michael Vitale for the report. Fix ORA-00942 with table that are not yet physically created and has no data. - Fix calling functions with same name declared in several packages. The one declared in current package now takes precedence. Thanks to Eric Delanoe for the report. - Change zero-length lob/long to undef workaround for a bug in DBD::Oracle with the ora_piece_lob option (used when no_lob_locator is enabled) where null values fetch as empty string for certain types. Thanks to Alice Maz for the patch. - Fix createPoint() spatial method issue. Thanks to jwl920919. - Fix comment on REPLACE_COLS directive. - Fix an other issue in transformation of TYPE x IS REF CURSOR. Thanks to Pavel Stehule for the report. - Fix an other case of broken declaration in triggers related to FOR cycle control variables. Thanks to Pavel Stehule for the report. - Fix broken declaration in triggers related to FOR cycle control variables with empty DECLARE section. - Fix other case of replacement by EXIT WHEN NOT FOUND. - Fix output of global_variables.conf file when OUTPUT_DIR is not set. Fix non replacement of global variables. - Remove some AUTHID garbage in procedure declaration generated by a migration. - Fix trigger name quoting with non supported character. Thanks to Michael Vitale for the report. - Fix use of nextval() in default value. - Fix alias append in from clause of the extract() function. Thanks to Narayanamoorthys for the report. - Disable direct export to partition for PostgreSQL 10 if directive PG_SUPPORTS_PARTITION is enabled, import must be done in the main table. - Do not export partitions of a materialized view. Thanks to Michael Vitale for the report. - Fix wrong replacement of keyword after END. Thanks to Pavel Stehule for the report. - Remove Oracle hints from queries, they are not supported and can break comments. Thanks to Pavel Stehule for the report. - Fix unstable transformation of TYPE x IS REF CURSOR. Thanks to Pavel Stehule for the report. - Fix data export failure when no table match the ALLOW/EXCLUDE filter. Thanks to threenotrump for the report. - Add missing search_path to FKEY dedicated file. Thanks to Michael Vitale for the report. - Apply default oject name exclusion to synonym. - Skip some PL/SQL translation in migration assessment mode. - Change default type for virtual column whit round() function. Thanks to Julien Rouhaud for the report. - Fix bug with EXIT WHEN command. Thanks to Pavel Stehule. - Fix an other wrong replacement of DECODE in UPDATE statement. Thanks to Pavel Stehule for the report. - Fix wrong replacement of DECODE. Thanks to Pavel Stehule. - Fix unwanted replacement of INTO STRICT when this is an INSERT statement. Thanks to Pavel Stehule for the report. - Fix potential regexp error with special character in outer join filters. Thanks to Adrian Boangiu for the report. - Fix parsing of PK from file. Thanks to Julien Rouhaud. - Fix parsing of FK from file. Thanks to Julien Rouhaud. - Fix count of unique and primary key in TEST export. Thanks to Liem for the report. - Fix reserved keyword rewrite using double quote when directive USE_RESERVED_WORDS is enabled. Thanks to Michael Vitale. - Remove EVALUATION CONTEXT object type from migration assessment. - Add QUEST_SL_% pattern to the list of table that must be excluded from export. - Fix data export when BFILE are translated as text. Thanks to Michael Vitale for the report. - Fix export of package when a comment is placed just before the AS/IS keyword. Thanks to Michael Vitale for the report. - Fix other cases of function call replacement when they are declared in different packages and one with OUT parameter and the other one with only IN parameters. Thanks to Eric Delanoe. - Fix inconsistent behavior of import_all script with -h and -d Under Linux: When -h not specified, script defaults to unix domain sockets for psql and localhost for perl (which may error depending on pg_hba.conf). Now defaults to more performing sockets. -d wasn't passing DB name to some psql calls where it's necessary. Thanks to BracketDevs for the patch. - Fix file handle close when compression is enabled. Thanks to Sebastian Albert for the report. - Add information about ora2pg behavior during data export to files when files already exists. Thanks to Michael Vitale. - Update readme to provide tar command for bzip2 file. Thanks to Tom Pollard for the patch - Fix unwanted FTS_INDEXES empty file and append unaccent extension creation if not exists. Thanks to Michael Vitale for the report. - Fix missing explicitly declared variable for cycle in trigger. Thanks to Pavel Stehule for the report. - Fix global type/cursor declaration doubled in package export. - Always translate Oracle SELECT ... INTO to SELECT ... INTO STRICT in plpgsql as Oracle seems to always throw an exception. Thanks to Pavel Stehule for the report. - Fix too much semicolons on end of function. Thanks to Pavel Stehule for the report. - Fix ALTER TABLE to set the owner when table is a foreign table. Thanks to Narayanamoorthys for the report. - Fix case of untranslated procedure call when there was parenthesis in the parameters clause. Thanks to Pavel Stehule for the report. - Fix broken variable declaration with name containing DEFAULT. Thanks to Pavel Stehule for the report. - Change query ORDER BY clause to view export query. - Fix missing replacement of quote_reserved_words function by new quote_object_name. Thanks to Sebastian Albert for the report. - Fix REPLACE_COLS replacement of column name in UNIQUE constraint definition. Thanks to Bernard Bielecki for the report. - Fix export of Oracle unlogged table that was exported as normal tables. - Fix regression in package function calls rewrite leading to append unwanted comma when replacing out parameters. Thanks to Pavel Stehule and Eric Delanoe for the report. - Fix removing of function name after END keyword. Thanks to Pavel Stehule for the report. - Fix bug in package function extraction. - Improve VIEW export by only looking for package function name and fix a bug that was including unwanted "system" package definition. Also fix a potential bad rewriting of function call. Thanks to Eric Delanoe for the report. - Fix an other case of missing PERFORM replacement. Thanks to Pavel Stehule for the report. - Fix remplacement of "EXIT WHEN cursor%NOTFOUND". Thanks to Pavel Stehule for the report. - Fix missing conversion of type in cast function. Thanks to Michael Vitale for the report. - Fix TO_NUMBER that is now translated as a cast to numeric to correspond to the default behavior in Oracle. Thanks to Pavel Stehule for the report. - Fix replacement of function call from different schema, especially in overloaded cases. - Remove OUT parameter from the argument list of function call. Thanks to Pavel Stehule for the report. - Fix wrong replacement in FOR ... IN loop inserting EXCLUDE in the statement. Thanks to Pavel Stehule for the report. - Translate Oracle proprietary VALUES without parenthesis with the proprietary syntax of POstgreSQL. Thanks to Pavel Stehule for the report. - Fix function header translation when a comment is present between closing parenthesis and the IS keyword. Thanks to Pavel Stehule for the report. - Fix RETURNS in autonomous transaction call when there is OUT parameters. Thanks to Pavel Stehule for the report. - Fix call to BMS_UTILITY.compile_schema() when COMPILE_SCHEMA is enable. Thanks to PAvel Stehule for the report. - Fix export of function and procedure with same name in different schema. Thanks to Pavel Stehule for the report. - Fix detection and replacement of global variable in package that was producing invalid code export. Fix progress bar on procedure export. - Fix regression in global variables default value export. - Rewrite multiprocess for procedure and function export to solve some performances issues. - Do not waste time trying to replace function call when it is not found in the current code. - Fix default value for FILE_PER_FUNCTION when parallel mode is enabled. - Output a fatal error with export type TABLE and multiple schema set to PG_SCHEMA when EXPORT_SCHEMA is enabled. - Fix replacement of function name with package prefix. - Fix documentation of PG_SCHEMA directive, especially on the use of a schema list. Thanks to Michael Vitale for the report. - Fix translation of INSTR() function. - Improve speed in function translation by not calling twice Ora2Pg::PLSQL::convert_plsql_code() on declare and code section. Thanks to Pavel Stehule for the profiling. - Fix unwanted replacement of SYSDATE, SYSTIMESTAMP and some other when they are part of variable or object name. Add rewrite of REF CURSOR during type translation. - Require support of LATERAL keyword for DATADIFF (Pg >= 9.3). Patch from Sebastian Albert. - Do not call replace_sdo_function(), replace_sdo_operator() and replace_sys_context() if the string SDO_ or SYSCONTEXT is not found. This might save some performances. - Remove the RETURNS clause when there is an OUT parameter PostgreSQL choose correct type by self. Thanks to Pavel Stehule for the report. - Add a note about performance improvement by updating stats on Oracle. Thanks to Michael Vitale for the report. - Remove newline characters in REVOKE statement when embedded in a comment. Thanks to Pavel Stehule for the report. - Fix replacement with PERFORM into package extracted from an Oracle database. Thanks to Eric Delanoe for the report. - Fix translation of call to function with out parameters. Thanks to Pavel Stehule for the report. - Fix case where call to procedure without parameter was not prefixed by PERFORM or when called in a exception statement. Thanks to Eric Delanoe for the report. - Add function quote_object_name to handle all cases where object name need to be double quoted (PRESERVE_CASE to 1, PostgreSQL keyword, digit in front or digit only and non supported character. Thanks to liemdt1811 for the report. - Add a note about RAW(n) column with "SYS_GUID()" as default value that is automatically translated to type of the column 'uuid' by Ora2Pg. - Remove old column count check to use char_length. Thanks to Alice Maz for the patch. - Fix some raise_application_error that was not replaced with a global rewrite of remove comments and text constants to solve some other issues like rewriting of package function call in dynamic queries. Thanks to PAvel Stehule for the report. - Fix cycle variable not generated for LOOP IN SELECT in trigger. Thanks to Pavel Stehule for the report. - Fix procedures with OUT parameters not processed in triggers. Thanks to Pavel Stehule for the report. - Remove other case where PERFORM must be or must not be inserted. - Remove case where PERFORM can be inserted. Thanks to Pavel Stehule and Eric Delanoe for the report. - Fix missing ; in some raise_application_error translation. Thanks to Pavel Stehule for the report. - Fix missing PERFORM in front of direct call to function and the rewrite of direct call to function with out parameters. Thanks to Eric Delanoe for the report. - Fix translation of rownum when the value is not a number. Thanks to Pavel Stehule for the report. - Fix missing space between cast and AS keyword. Thanks to Pavel Stehule for the report. - Fix translation of views and add support to comment inside views. Thanks to Pavel Stehule for the report. - Fix removing of AS after END keyword. Thanks to Pavel Stehule for the report. - Fix type in CAST clause not translated to PostgreSQL type. Thanks to Pavel Stehule for the report. - Treat citext type as text. Thanks to Tomasz Wrobel for the patch. - Fix packages migration assessment that was broken with parser rewriting on package extraction. - Rewrite parsing of PL/SQL packages to better handle package specification and especially types and global variables from this section. - Fix raise_application_error translation by removing extra boolean parameter. - Improve comments processing. - Fix package function name replacement adding a dot before package name. Thanks to Eric Delanoe for the report. - Add collect of functions/procedures metadata when reading DDL from file. - Fix replacement of function prefixed with their schema name. Thanks to Eric Delanoe for the report. - Try to minimized comment placeholders by aggregating multiline comments. - Remove new line character from _get_version() output. - Fix ENABLE_MICROSECOND test condition on NLS_TIMESTAMP_TZ_FORMAT setting. Thanks to Didier Sterbecq for the report. - Fix another issue with Oracle 8i and table size extraction. - Fix query to show column information on Oracle 8i - Fix query to look for virtual column on Oracle 8i - Fix query to list all table by size on Oracle 8i - Prevent ora2pg to look for external table definition in Oracle 8i. - Fix a regression on timestamp format setting for Oracle 8i. - Fix some regression on queries with Oracle 8i. Thanks to Didier Sterbecq for the report. - Add a function to collect metadata of all functions. - Don't create empty partition index file when there's no partition. - Fix wrong translation in OPEN ... FOR statement. Thanks to Eric Delanoe for the report. - Fix call of method close() on an undefined value. Thanks to Eric Delanoe for the report. - Fix partition data export issues introduced with previous patches. - Fix unterminated IF / ELSIF block in subpartition export. - Fix subpartition export. Thanks to Maurizio De Giorgi for the report. - Force DATA_LIMIT default value to 2000 on Windows OS instead of 10000 to try to prevent constant OOM error. Thanks to Eric Delanoe for the report. - Fix default partition table that was not used PREFIX_PARTITION. Thanks to ssayyadi for the report. - Limit datetime microsecond format to micro second (.FF6) as the format can be FF[0..9] and PostgreSQL just have FF[0..6] - Add to_timestamp_tz Oracle function translation. Thanks to Eric Delanoe for the feature request. - Fix custom data type replacement in function code. Thanks to Pavel Stehule for the report. - Fix non working INPUT_FILE configuration directive when action is QUERY. Thanks to Eric Delanoe for the report. - Fix unwanted global variable implicit declaration to handle autonomous transaction parameters. Thanks to Eric Delanoe for the report. - Fix call to dblink in function with PRAGMA AUTONOMOUS_TRANSACTION and no arguments. Thanks to Eric Delanoe for the report. - Fix package constant translation. Thanks to Eric Delanoe. - Fix unwanted alias on join syntax. Thanks to Eric Delanoe for the report. - Fix regression on dbms_output.put* translation. Thanks to Eric Delanoe for the report. - Fix handling of comments in statements to try to preserve them at maximum in the outer join rewriting. - Do not declare variable when it is an implicit range cursor, it do not need to be declared. - Export implicit variable in FOR ... IN ... LOOP as an integer if it don't use a select statement and export it as a RECORD when a statement is found. Thanks to Eric Delanoe and Pavel Stehule for the report. - Reduce migration assessment weight for CONNECT BY. - Fix derived table pasted two times in from clause. Thanks to Pavel Stehule for the report. - Fix some other unexpected ";" in function code. Thanks to Pavel Stehule for the report. - Remove %ROWTYPE in return type of function. Thanks to Pavel Stehule for the report. - Fix doubled AND in expression when a parenthesis is in front after rewriting. Thanks to Eric Delanoe for the report. - Fix unexpected ";" in function after post-body END when a comment is present. Thanks to Eric Delanoe for the report. - Fix unexpected ";" in some function variable declaration when a comment is at end of the declare section. Thanks to Eric Delanoe for the report. - Remove %ROWTYPE in function that have not been replaced with RECORD for cursor declaration. Thanks to Eric Delanoe for the report. - Fix removing of WHEN keyword after END. Thanks to Pavel Stehule for the report. - Fix missing table name with alias in from clause due to comments in the clause. I have also merge right and left outer join translation function into a single one, most of the code was the same. - Fix output order of outer join. Thanks to Pavel Stehule for the report. - Fix untranslated outer join in nested sub query. Thanks to Pavel Stehule for the report. - Rewrite again the decode() translation as a single function call for all replacement before any other translation. - Append table filter to check constraints extraction. Thanks to danghb for the report. - Fix issue with parenthesis around outer join clause. Thanks to Pavel Stehule for the report. - Move remove_text_constant_part() and restore_text_constant_part() function into the main module. - Include decode() replacement in recursive function call. Thanks to Pavel Stehule for the report. - Prevent removing of parenthesis on a sub select. Thanks to Pavel Stehule for the report. - Fix missing table exclusion/inclusion in column constraint export. Thanks to danghb for the report. - Fix an alias issue in view parsed from file. - Fix parsing of view from file when no semi comma is found. - Remove FROM clause without alias from migration assessment. - Fix order of outer join during translation. Thanks to Pavel Stehule for the report. - Fix case of missing alias on subquery in FROM clause. Thanks to Pavel Stehule for the report. - Fix missing alias replacement in nested subqueries. Thanks to Pavel Stehule for the report. - Fix wrong addition of aliases to using() in join clause - Fix nested decode replacement producing invalid CASE expression. Thanks to Pavel Stehule for the report. - Append aliases to subqueries in the from clause that do not have one. Thanks to Pavel Stehule for the report. 2017 02 17 - v18.1 This release fix several issues reported on outer join translation thanks to the help of Pavel Stehule and reapply the commit on virtual column export that was accidentally removed from v18.0. It also adds several new features: - Remove CHECK constraints for columns converted into boolean using REPLACE_AS_BOOLEAN column. - Oracle function are now marked as stable by default as they can not modify data. Two new configuration directives have been added: - DATE_FUNCTION_REWRITE: by default Ora2pg rewrite add_month(), add_year() and date_trunc() functions set it to 0 to force Ora2Pg to not translate those functions if translated code is broken. - GRANT_OBJECT: when exporting GRANT you can now specify a comma separated list of objects which privileges must be exported. Default is to export privileges for all objects. For example set it to TABLE if you just want to export privilege on tables. and a new command line option: - Add -g | --grant_object command line option to ora2pg to be able to extract privilege from the given object type. See possible values with GRANT_OBJECT configuration directive. Here is the complete list of changes: - Remove empty output.sql file in current directory with direct data import. Thanks to kuzmaka for the report. - Fix shell replacement of $$ in function definition in Makefile.PL embedded configuration file. Thanks to kuzmaka for the report. - Fix shell replacement of backslash in Makefile.PL embedded configuration file. Thanks to kuzmaka for the report. - Add warning level to virtual column notice. - Fix comment in where clause breaking the outer join association. Thanks to Pavel Stehule for the report. - Add parsing and support of virtual column from DDL file. - Reapply commit on virtual column export that was accidentally removed in commit d5866c9. Thanks to Alexey for the report. - Fix mix of inner join and outer join not translated correctly. Thanks to Pavel Stehule for the help to solve this issue. - Fix additional comma in column DEFAULT value from DDL input file. Thanks to Cynthia Shang for the report. - Fix comments inside FROM clause breaking translation to ANSI outer joins. Thanks to Pavel Stehule for the report. - Fix replacement of sdo_geometry type into function. Thanks to Saber Chaabane for the report. - Fix subquery in outer join clause. Thanks to Saber Chaabane for the report. - Fix duplicated subqueries placeholder in the from clause. Thanks to Saber Chaabane for the report. - Fix replacement of subquery place older during outer join rewrite. Thanks to Saber Chaabane for the report. - Add DATE_FUNCTION_REWRITE configuration directive. By default Ora2pg rewrite add_month(), add_year() and date_trunc() functions set it to 0 to force Ora2Pg to not translate those functions if translated code is broken. Thanks to Pavel Stehule for the feature request. - Do not report error when -g is used but action is not GRANT. Thanks to Shane Jimmerson for the report. - Oracle function can not modify data, only procedure can do that, so mark them as stable. Thanks to Pavel Stehule for the report. - Missed some obvious combination like upper/lower case or no space after AND/OR on outer join parsing and some other issues. - Add missing call to extract_subqueries() recursively. Thanks to Pavel Stehule for the report. - Add full support of outer join translation in sub queries. - Add translation of mixed inner join and Oracle outer join. Thanks to Pavel Stehule for the report. - Fix missing space between keyword AS and END from the decode() transformation. Thanks to Pavel Stehule for the report. - Fix parsing of outer join with UNION and translation to left join. Thanks to Pavel Stehule for the report. - Remove CHECK constraints for columns converted into boolean using REPLACE_AS_BOOLEAN column. Thanks to Shane Jimmerson for the feature request. - Fix regression on SQL and PLSQL rewrite when a text constant contained a semi-comma. - Add the GRANT_OBJECT configuration directive. When exporting GRANT you can specify a comma separated list of objects for which the privileges will be exported. Default is export for all objects. Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM and DIRECTORY. Only one object type is allowed at a time. For example set it to TABLE if you just want to export privilege on tables. You can use the -g option to overwrite it. When used this directive prevent the export of users unless it is set to USER. In this case only users definitions are exported. - Add the -g | --grant_object command line option to ora2pg to be able to extract privilege from the given object type. See possible values with GRANT_OBJECT configuration directive. - Improve replacement of ROWNUM by LIMIT+OFFSET clause. - Fix extra semi-colon at end of statement. - Override ora2pg.spec with Devrim's one but with String::Random removing as it is no more used. 2017 01 29 - v18.0 This new major release adds several new useful features and lot of improvements. * Automatic rewrite of simple form of (+) outer join Oracle's syntax. This major feature makes Ora2Pg become the first free tool that is able to rewrite automatically (+) outer join in command line mode. This works with simple form of outer join but this is a beginning. * Add export of Oracle's virtual column using a real column and a trigger. * Allow conversion of RAW/CHAR/VARCHAR2 type with precision in DATA_TYPE directive. Useful for example to transform all RAW(32) or VARCHAR2(32) columns into PostgreSQL special type uuid. * Add export NOT VALIDATED state from Oracle foreign keys and check constraints into NOT VALID constraints in PostgreSQL. * Replace call to SYS_GUID() with uuid_generate_v4() by default. * Add "CREATE EXTENSION IF NOT EXISTS dblink;" before an autonomous transaction or "CREATE EXTENSION IF NOT EXISTS pg_background;". * Major rewrite of the way Ora2Pg parse PL/SQL to rewrite function calls and other PL/SQL to plpgsql replacement. There should not be any limitation in rewriting when a function contains a sub query or an other function call inside his parameters. * Refactoring of ora2pg to not requires any dependency other than the Perl DBI module by default. All DBD drivers are now optionals and ora2pg will expect an Oracle DDL file as input by default. * Add export of Oracle's global variables defined in package. They are exported as user defined custom variables and available in a session. If the variable is a constant or have a default value assigned at declaration, ora2pg will create a new file with the declaration (global_variables.conf) to be included in the main configuration file postgresql.conf file. * Create full text search configuration when USE_UNACCENT directive is enabled using the auto detected stemmer or the one defined in FTS_CONFIG. For example: CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; CREATE INDEX place_notes_cidx ON places USING gin(to_tsvector('fr', place_notes)); Changes and incompatibilities from previous release: * FTS_INDEX_ONLY is now enabled by default because the addition of a column is not always possible and not always necessary where a simple function-based index is enough. * Remove use to setweigth() on single column FTS based indexes. * Change default behaviour of Ora2Pg in Full Text Search index export. A new command line option and some configuration directive have been added: * Option -D | --data_type to allow custom data type replacement at command line like in configuration file with DATA_TYPE. * UUID_FUNCTION to be able to redefined the function called to replace SYS_GUID() Oracle function. Default to uuid_generate_v4. * REWRITE_OUTER_JOIN to be able to disable the rewriting of Oracle native syntax (+) into OUTER JOIN if rewritten code is broken. * USE_UNACCENT and USE_LOWER_UNACCENT configuration directives to use the unaccent extension with pg_trgm with the FTS indexes. * FTS_INDEX_ONLY, by default Ora2Pg creates an extra tsvector column with a dedicated triggers for FTS indexes. Enable this directive if you just want a function-based index like: CREATE INDEX ON t_document USING gin(to_tsvector('pg_catalog.english', title)); * FTS_CONFIG, use this directive to force the text search stemmer used with the to_tsvector() function. Default is to auto detect the Oracle FTS stemmer. For example, setting FTS_CONFIG to pg_catalog.english or pg_catalog.french will override the auto detected stemmer. There's also lot fixes of issues reported by users from the past two months, here is the complete list of changes: - Fix return type in function with a single inout parameter and a returned type. - Prevent wrong rewrite of empty as null when a function is used. Thanks to Pavel Stehule for the report. - Add the UUID_FUNCTION configuration directive. By default Ora2Pg will convert call to SYS_GUID() Oracle function with a call to uuid_generate_v4 from uuid-ossp extension. You can redefined it to use the gen_random_uuid function from pgcrypto extension by changing the function name. Default to uuid_generate_v4. Thanks to sjimmerson for the feature request. - Add rewrite of queries with simple form of left outer join syntax (+) into the ansi form. - Add new command line option -D | --data_type to allow custom data type replacement at command line like in configuration file with DATA_TYPE. - Fix type in ROWNUM replacement expression. Thanks to Pavel Stehule for the report. - Add replacement of SYS_GUID by uuid_generate_v4 and allow custom rewriting of RAW type. Thanks to Nicolas Martin for the report. - Fix missing WHERE clause in ROWNUM replacement with previous patch thanks to Pavel Stehule for the report. - Fix ROWNUM replacement when e sub select is used. Thanks to Pavel Stehule for the report. - Fix wrong syntax in index creation with DROP_INDEXES enabled. Thanks to Pave Stehule for the report. - Remove replacement of substr() by substring() as PostgreSQL have the substr() function too. Thanks to Pavel Stehule for the report. - Move LIMIT replacement for ROWNUM to the end of the query. Thanks to Pavel Stehule for the report. - Fix text default value between parenthesis in table declaration. Thanks to Pavel Stehule for the report. - Fix return type when a function have IN/OUT parameter. Thanks to Pavel Stehule for the report. - Mark uuid type to be exported as text. Thanks to sjimmerson for the report. - Add EXECUTE to open cursor with like "OPEN var1 FOR var2;". Thanks to Pavel Stehule for the report. - Fix replacement of local type ref cursor. Thanks to Pavel Stehule for the report. - Add EXECUTE keyword to OPEN CURSOR ... FOR with dynamic query. Thanks to Pavel Stehule for the report. - Fix case sensitivity issue in FOR .. IN variable declaration replacement. Thanks to Pavel Stehule for the report. - Fix wrong replacement of cast syntax ::. Thanks to Pavel Stehule for the report. - Reactivate numeric cast in call to round(value,n). - Close main output data file at end of export. - Add virtual column state in column info report, first stage to export those columns as columns with associated trigger. - Fix unwanted replacement of REGEXP_INSTR. Thanks to Bernard Bielecki for the report. - Allow rewrite of NUMBER(*, 0) into bigint or other type instead numeric(38), just set DATA_TYPE to NUMBER(*\,0):bigint. Thanks to kuzmaka for the feature request. - Export partitions indexes into PARTITION_INDEXES_....sql separate file named. Thanks to Nicolas Martin for the feature request. - Fix fatal error when schema CTXSYS does not exists. Thanks to Bernard Bielecki for the report. - Fix missing text value replacement. Thanks to Bernard Bielecki for the report. - Fix type replacement in declare section when the keyword END was present into a variable name. - Export NOT VALIDATED Oracle foreign key and check constraint as NOT VALID in PostgreSQL. Thanks to Alexey for the feature request. - Add object matching of regex 'SYS_.*\$' to the default exclusion list. - Fix UTF8 output to file as the open pragma "use open ':utf8';" doesn't works in a global context. binmode(':encoding(...)') is used on each file descriptor for data output. - Improve parsing of tables/indexes/constraints/tablespaces DDL from file. - Improve parsing of sequences DDL from file. - Improve parsing of user defined types DDL from file. - Export Oracle's TYPE REF CURSOR with a warning as not supported. - Replace call to plsql_to_plpgsql() in Ora2Pg.pm by a call to new function convert_plsql_code(). - Move export of constraints after indexes to be able to use USING index in constraint creation without error complaining that index does not exists. - Add "CREATE EXTENSION IF NOT EXISTS dblink;" before an autonomous transaction or "CREATE EXTENSION IF NOT EXISTS pg_background;". - Improve parsing of packages DDL from file. - When a variable in "FOR varname IN" statement is not found in the DECLARE bloc, Ora2Pg will automatically add the variable to this bloc declared as a RECORD. Thanks to Pavel Stehule for the report. - Major rewrite of the way Ora2Pg parse PL/SQL to rewrite function calls and other PL/SQL to plpgsql replacement. There should not be limitation in rewriting when a function contains a sub query or an other function call inside his parameters. - Fix unwanted SELECT to PERFORM transformation inside literal strings. Thanks to Pavel Stehule for the report. - Fix bug in DEFAULT value rewriting. Thanks to Pavel Stehule for the report. - Fix replacement of DBMS_OUTPUT.put_line with RAISE NOTICE. - Reset global variable storage for each package. - Improve comment parsing in packages and prevent possible infinite loop in global variable replacement. - Add the REWRITE_OUTER_JOIN configuration directive to be able to disable the rewriting of Oracle native syntax (+) into OUTER JOIN if it is broken. Default is to try to rewrite simple form of right outer join for the moment. - Export types and cursors declared as global objects in package spec header into the main output file for package export. Types and cursors declared into the package body are exported into the output file of the first function declared in this package. - Globals variables declared into the package spec header are now identified and replaced into the package code with the call to user defined custom variable. It works just like globals variables declared into the package body. - Add auto detection of Oracle FTS stemmer and disable FTS_CONFIG configuration directive per default. When FTS_CONFIG is set its value will overwrite the auto detected value. - Create full text search configuration when USE_UNACCENT directive is enabled using the auto detected stemmer or the one defined in FTS_CONFIG. For example: CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; CREATE INDEX place_notes_cidx ON places USING gin(to_tsvector('fr', place_notes)); - Remove CONTAINS(ABOUT()) from the migration assessment, there no additional difficulty to CONTAINS rewrite. - Add ANYDATA to the migration assessment keyword to detect. - Allow conversion of CHAR/VARCHAR2 type with precision in DATA_TYPE directive. For example it's possible to transform all VARCHAR2(32) columns only into PostgreSQL special type uuid by setting: DATA_TYPE VARCHAR2(32):uuid Thanks to sjimmerson for the feature request. - Update year in copyrights - Fix creation of schema when CREATE_SCHEMA+PG_SCHEMA are defined. - Fix renaming of temporary file when exporting partitions. - Move MODIFY_TYPE to the type section - Update documentation about globals variables. - Add export of Oracle's global variables defined in package. They are exported as user defined custom variables and available in a session. Oracle variables assignment are exported as call to: PERFORM set_config('pkgname.varname', value, false); Use of these variable in the code is replaced by: current_setting('pkgname.varname')::global_variables_type; the variable type is extracted from the pacjkage definition. If the variable is a constant or have a default value assigned at declaration, ora2pg will create file global_variables.conf with the definition to include in postgresql.conf file so that their values will already be set at database connection. Note that the value can always modified by the user so you can not have exactly a constant. - Fix migration assessment of view. - Remove call to FROM SYS.DUAL, only FROM DUAL was replaced. - Replace call to trim into btrim. - Improve rewrite of DECODE when there is function call inside. - Add function replace_right_outer_join() to rewrite Oracle (+) right outer join. - Improve view migration assessment. - Create a FTS section in the configuration file dedicated to FTS control. - Add USE_UNACCENT and USE_LOWER_UNACCENT configuration directives to use the unaccent extension with pg_trgm. - Do not create FTS_INDEXES_* file when there is no Oracle Text indexes. - Update query test score when CONTAINS, SCORE, FUZZY, ABOUT, NEAR keyword are found. - Remove use to setweigth() on single column FTS based indexes. Thanks to Adrien Nayrat for the report. - Update documentation on FTS_INDEX_ONLY with full explanation on the Ora2Pg transformation. - Refactoring ora2pg to not requires any dependency other than the Perl DBI module by default. All DBD drivers are now optionals and ora2pg will expect to received an Oracle DDL file as input by default. This makes easiest packaging or for any distribution that can not build a package because of the DBD::Oracle requirement. DBD::Oracle, DBD::MySQL and DBD::Pg are still required if you want Ora2Pg to migrate your database "on-line" but they are optional because Ora2Pg can also convert input DDL file, this is the default now. Thanks to Gustavo Panizzo for the feature request and the work on Debian packaging. - Remove String::Random dependency in rpm spec file, it is no used even if it was mentioned into a comment. - Exclude internal Oracle Streams AQ JMS types from the export. Thanks to Joanna Xu for the report. - Fix some other spelling issues. Thanks to Gustavo Panizzo for the patch. - Fix some spelling errors. Thanks to Gustavo Panizzo for the patch. - Revert patch 697f09d that was breaking encoding with input file (-i). Thanks to Gary Evans for the report. - Add two new configuration directive to control FTS settings, FTS_INDEX_ONLY and FTS_CONFIG. 2016 11 17 - v17.6 This release adds several new features: * Adds export of Oracle Text Indexes into FTS or pg_trgm based indexes, * Add export of indexes defined on materialized views * Allow export of materialized views as foreign tables when export type is FDW. * Add replacement of trim() by btrim(). Two new configuration directives have been added: * USE_INDEX_OPCLASS: when value is set to 1, this will force Ora2Pg to export all indexes defined on varchar2() and char() columns using *_pattern_ops operators. If you set it to a value greater than 1 it will only change indexes on columns where the character limit is greater or equal than this value. * CONTEXT_AS_TRGM: when enabled it forces Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes and CTXCAT indexes using pg_trgm. Some time using pg_trgm based indexes is enough. There's also some fixes of issues reported by users, here is the complete list of changes: - Fixed non-use of custom temp_dir (-T). Thanks to Sebastian Albert for the patch. - Make export of FTS indexes from materialized view work as for tables. - Fix drop of indexes during export of data when DROP_INDEXES is enabled. - Remove double quote in function and procedure name from an input file to avoid creating a file with double quote in its name. - Fix export of unique index associated to a primary key. - Move OPTION (key "yes") of FDW table before NOT NUL constraint and default clause. - Fix some encoding issue during data export into file. - Rename FTS indexes prefix output file into FTS_INDEXES and export CTXCAT Oracle indexes as GIN pg_trgm indexes instead of FTS indexes. - Add export of indexes of type CTXCAT as FTS indexes. - Export triggers and update order for FTS indexes to separate file prefixed with FTS_INDEXES. - Exclude from export synonyms starting with a slash that correspond to incomplete deleted synonyms. Thanks to Nouredine Mallem for the report. - Add export of indexes defined on materialized views. Thanks to Nouredine Mallem for the report. - Fix export of foreign key and FTS indexes when looking at dba_* tables and multiple different schemas have the same fk or context indexes definition. Thanks to Nouredine Mallemfor the patch. - Fix export of CONTEXT or FULLTEXT Oracle index into PostgreSQL FTS with trigger and initial update statement. - Add configuration directive USE_INDEX_OPCLASS to force Ora2Pg to export all indexes defined on varchar2() and char() columns using those operators. A value greater than 1 will only change indexes on columns where the character limit is greater or equal than this value. - Fix FDW export of mysql tables. Thanks to yafeishi for the report. - Fix decode() rewrite. Thanks to Jean-Yves Julliot for the report. - Fix regression introduced into the export of NUMBER to integer like PG types. - Show partition name in progress bar instead of main table name. 2016 10 20 - v17.5 This is a maintenance release to fix several issues reported by users. There is also some major improvement and new feature. There is a new configuration directive or change default behavior: * Fix export of CLOBs and NCLOB that was truncated to 64 Kb. * PG_BACKGROUND : when enabled autonomous transactions will be built using Robert Haas extension pg_background instead of dblink. Default is to still used dblink as pg_background is available only for PostgreSQL >= 9.5. * All Perl I/O now use the open pragma instead of calling method binmode(). This will force input and output to utf8 using the Perl pragma: use open ':encoding(utf8)'; when configuration directive BINMODE is not set or NLS_LANG is set to UTF8. * Ora2Pg will now export empty lob as empty string instead of NULL when the source column has NOT NULL constraint and that directive EMPTY_LOB_NULL is not activated. * Improve and fix progress bar especially when using JOBS/-J option. * Allow LOAD action to apply all settings defined in the input file on each opened session, this allow to use LOAD with export schema enabled. If settings are not set in the input file encoding and search_path is set from the ora2pg configuration settings. * NUMBER(*,0) is now exported as numeric(38) as well as a NUMBER with DATA_SCALE set to 0, no DATA_PRECISION and a DATA_LENGTH of 22. The last correspond to Oracle type INTEGER or INT. * Allow conversion of type with precision in DATA_TYPE directive. For example it is possible to transform all NUMBER(12,2) only into numeric(12,2) by escaping the comma. Example: DATA_TYPE NUMBER(12\,2):numeric(12\,2);... * Write data exported into temporary files (prefixed by tmp_) and renamed them at end of the export to be able to detect incomplete export and override it at next export. * Add export of type created in package declaration. * Export foreign key when the referenced table is not in the same schema. * Enabled by default PG_SUPPORTS_CHECKOPTION assuming that your Pg destination database is at least a 9.4 version. * Add 12 units to migration assessment report per table/column conflicting with a reserved word in PostgreSQL to reflect the need of code rewriting. * Output a warning when a column has the same name than a system column (xmin,xmax,ctid,etc.) * Replace SYSDATE by a call to clock_timestamp() instead of a call to LOCALTIMESTAMP in plpgsql code. * Add missing documentation about DISABLE_PARTITION directive used to not reproduce partitioning into PostgreSQL and only export partitioned data into the main table. * Show partition name in progress bar instead of main table name. Here is the complete list of other changes: - Fix broken parallel table export (-P option). - Fix export of CLOBs and NCLOB that was truncated to 64Kb. Thanks to Paul Mzko for the patch. - Fix database handle in error report. - Fix use of wrong database handle to set search_path. Thanks to Paul Mzko for the report. - Ora2pg doesn't export schema ForeignKey constraint when connected as different DBA user. Thanks to Paul Mzko for the patch. - Fix Perl I/O encoding using open pragma instead of calling method binmode(). Thanks to Gary Evans for the report. - Force input to utf8 using Perl pragma: use open ':encoding(utf8)'; when BINMODE is not set or NLS_LANG is UTF8. - Force ora2pg to export empty lob as empty string instead of NULL when the source column has a NOT NULL constraint and directive EMPTY_LOB_NULL is not activated. Thanks to Valeriy for the report. - Fix missing CASCADE attribute on fkey creation during data export when DROP_FKEY was enabled. Thanks to ilya makarov for the report. - Fix issue on converting NUMBER(*,0) to numeric, should be ported to numeric(38). Thanks to ilya makarov for the report. - Correct query for ForeignKey export from oracle. Thanks to ilya makarov for the patch. - Fix schema change in direct import of data to PostgreSQL. - Change query for foreign key extraction to keep the column order. Thanks to ilya makarov for the report. - Write data exported into temporary files (prefixed by tmp_) and renamed them at end of the export to be able to detect incomplete export and override it at next export. Thanks to Paul Mkzo for the feature request. - Fix infinite loop in blob extraction when error ORA-25408 occurs during ora_lob_read() call. Thanks to Paul Mzko for the report. - Fix order of columns in foreign keys definition. Thanks to ilya makarov for the report. - Fix export of partition by range on multicolumn. Thanks to Rupesh Admane for the report. - Update reserved keywords list. Thanks to Nicolas Gollet for the report. - Add ON DELETE NO ACTION on foreign key creation (DROP_FKEY) to obtain the same output than during constraints export. - Fix export of foreign key that was duplicating the columns in both part, local and foreign. Thanks to ilya makarov for the report. - Remove call to to_char(datecol, format) when exporting date and timestamp. This formating is no more needed as we are now forcing NLS_DATE_FORMAT and NLS_TIMESTAMP_FORMAT when opening a connection to Oracle using: ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS and ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS This may result on some speed improvment during data export. - Fix parsing of packages from input file. - Add export of type created in package declaration. Thanks to dezdechado for the report. - Fix converting of procedures with out arguments. Thanks to dezdechado for the report. - Update documentation about project management. - Fix replacement of = NULL by IS NULL in update statement. Thanks to dezdechado for the report. - Fix parsing of trigger from file that was broken and new line removed. Thanks to dezdechado for the report. - Fix erasing of quotes from text in triggers. Thanks to dezdechado for the report. - Fix "return new" on trigger function when there is exception. Thanks to dezdechado for the report and solution. - Fix conversion of INTEGER and INT into numeric(38) instead of numeric without precision. Thanks to dezdechado for the report. - Fix export of foreign key when the referenced table is not in the same schema. Thanks to Juju for the report. - Fix ddl create schema when EXPORT_SCHEMA and CREATE_SCHEMA are enabled but no schema is specified. - Fix export of NCHAR that was converted as char but was loosing its length definition. Thanks to lgerlandsen for the report. - Fix parsing of views using WITH statements. Thank to dezdechado for the report. - Fix case that breaks views/triggers definition when a semicolon is encountered in a string value when reading definition from file. Thanks to dezdechado for the report. - Fix included/excluded of sequences when using ALLOW/EXCLUDE directives. Thanks to Roman Sindelar for the report. - prepare options modified with some escaping improvements. Thanks to ioxgrey for the patch. - It seems that for a NUMBER with a DATA_SCALE set to 0, no DATA_PRECISION and a DATA_LENGTH of 22 in ALL_TAB_COLUMNS, Oracle use a NUMBER(38) instead. This correspond to Oracle type INTEGER or INT. I don't really understand the reason of this behavior, why not just using a data length of 38? ALL_TAB_COLUMNS and Ora2Pg reports a data length of 22, now Ora2Pg will report NUMBER(38) like the resulting type of the DESC command. The following Oracle table: CREATE TABLE TEST_TABLE ( FIELD_1 INTEGER, FIELD_2 NUMBER ); will be exported as follow by Ora2Pg: [1] TABLE TEST_TABLE (owner: HR, 0 rows) FIELD_1 : NUMBER(38) => numeric FIELD_2 : NUMBER(22) => bigint Oracle data type INTEGER and INT will be exported as numeric by default instead of an integer. - Fix parsing of function/procedure from file with comments after BEGIN statement. - Remove DEFERRABLE INITIALLY DEFERRED from CHECK constraints when parsed from file. Thanks to Felipe Lavoura for the report. - Fix double parenthesis in index definition when parsing index creation from file. Thanks to Felipe Lavoura for the report. - Fix parsing of COMMENT from file. - Fix undetected native Oracle type bug. Thanks to kvnema for the report. - Fix unwanted text formatting with bind value in INSERT action with direct import to PostgreSQL. Thanks to Oleg for the report. - Fix inversion of UPDATE_RULE and DELETE_RULE in foreign key creation for MySQL export. Thanks to Sebastian Albert for the report. - Update documentation about DEFER_FKEY and DROP_FKEY to report the new behavior. - Remove call to SET CONSTRAINTS ALL DEFERRED with direct import. - Fix use of NULL value in bind parameter that should be undefined (INSERT export mode only). Thanks to Oleg barabaka for the report. - Remove replacement of direct call to functions with PERFORM, there is too much false positive. Thanks to dezdechado for the reports. - Fix a typo in SYSDATE replacement. Thank to dezdechado for report. - Remove rewrite of concatenation in RAISE EXCEPTION. Thanks to dezdechado for the report. - Fix replacement of raise_application_error() when first argument is a variable. Thanks to dezdechado for the report. - Fix wrong insertion of PERFORM before some function calls. Thanks to dezdechado for the report. - Replace SYSDATE by a call to clock_timestamp() instead of call to LOCALTIMESTAMP in plpgsql code. Thanks to aleksaan for the report. - Allow use of comma for object name list separator instead of space as workaround on Window OS. - Fix documentation about MODIFY_TYPE. Thanks to Nicolas Gollet for the report. - Add missing documentation about DISABLE_PARTITION directive used to not reproduce partitioning into PostgreSQL and only export partitioned data into the main table. Thanks to Nicolas Gollet for the report. - Add information about how to export LONG RAW data type. They need to be exported as BLOB before into Oracle to be exported as BYTEA. - Fix case where select was wrongly replaced by perform in INSERT INTO with SELECT statement. Thanks to dezdechado for the report. - Fix links to ora2pg presentation. Thanks to Daniel Lenski for the patch. - Fix input parameters after one with a default value must also have defaults. Thanks to v.agapov fot the patch. - Fix debug mode that was interromping the last running table dump. Thanks to calbiston for the report. 2016 04 21 - v17.4 Errata in first release attempt. - Fix previous patch that does not handle blob case but just clob - Forgot to change back the query when EMPTY_LOB_NULL is not activated. - Put parenthesis around AT TIME ZONE expression This is a maintenance release to fix several issues reported by users. There is also some major data export speed improvement thanks to the work of PostgreSQL Pro and a new RPM spec file provided by Devrim Gunduz to be able to build RPM package for Ora2Pg. There is a new configuration directive: - EMPTY_LOB_NULL: when enabled force empty_clob() and empty_blob() to be exported as NULL instead as empty string. Here is the complete list of other changes: - Add EMPTY_LOB_NULL directive to force empty_clob() and empty_blob() to be exported as NULL instead as empty string. This might improve data export speed if you have lot of empty lob. Thanks to Alex Ignatov for the report. - Fix import_all.sh script to import grant and tablespace separately as postgres user and just after indexes and constraints creation. - Add parsing of tablespace from "alter table ... add constraint" with DDL input file. Thanks to Felipe Lavoura. - Remove --single-transaction in import_all.sh script with TABLESPACE import. Thanks to Guillaume Lelarge for the report. - Fix Makefile.PL to used with latest spec file from Devrim Gunduz and following the advice of calbiston. - Update spec file to v17.6 and latest change to Makefile.PL - Replace ora2pg.spec by postgressql.org spec file by Devrim Gunduz. - Generate man page to avoids rpmbuild error. - Fix Windows install. Thanks to Lorena Figueredo for the report. - Remove "deferrability" call for mysql foreign keys. Thanks to Jean-Eric Cuendet for the report. - Fix issue in restoring foreign key for mysql data export. Thanks to Jean-Eric Cuendet for the report. - Remove connection test to PostgreSQL instance as postgres or any superuser in import_all.sh - Fix creation of configuration directory. - Fix Makefile to dissociate CONFDIR and DOCDIR from PREFIX or DESTDIR. Thanks to Stephane Schildknecht for the report. - Fix date_trunc+add_month replacement issue. Thanks to Lorena Figueredo for the report. - Do not replace configuration directory in scripts/ora2pg if this is a RPM build. Thanks to calbiston for the report. - Return empty bytea when a LOB is empty and not NULL. - Regular expressions and conditions checks improvement in method format_data_type() to make it a bit faster on huge tables. Thanks to Svetlana Shorina for the patch. - Fix INSERT using on the fly data import with boolean values. Thanks to jecuendet for the report. - Allow MySQL data type to be converted into boolean. Thanks to jecuendet for the report. - Fix export of BIT mysql data type into bit bit varying. Thanks to jecuendet for the report. - Fix call to escape_copy/escape_insert function call. 2016 03 26 - v17.3 This release fix two regressions introduced in latest release. * Fix major bug in data export. Thanks to Frederic Guiet for the report. * Fix another regression with character data that was not escaped. Thanks to Frederic Guiet for the report. 2016 03 24 - v17.2 This is a maintenance release to fix several issues reported in new LOB extraction method. There is also some feature improvement: * Allow NUMBER with precision to be replaced as boolean. * Allow full relocation of Ora2Pg installation using for example: perl Makefile.PL DESTDIR=/opt/ora2pg Here is the complete list of other changes: - Allow NUMBER with precision to be replaced as boolean. Thanks to Silvan Auer for the report. - Force empty LOB to be exported as NULL when NO_LOB_LOCATOR is activated to have the same behavior. - Fix case where a LOB is NULL and ora2pg reports error : DBD::Oracle::db::ora_lob_read: locator is not of type OCILobLocatorPtr LOB initialised with EMPTY_CLOB() are also exported as NULL instead of \\x - Fix replacement with PERFORM after MINUS. Thanks to Stephane Tachoires for the report. - Comment DBMS_OUTPUT.ENABLE calls. Thanks to Stephane Tachoire for the report. - Fix wrong replacement of SELECT by PERFORM after EXCEPT. Thanks to Stephane Tachoire for the report. - Apply ORACLE_COPIES automatic predicate on custom queries set with REPLACE_QUERY if possible. Thanks to pawelbs for the report. - Fix install of ora2pg.conf file in /etc/ instead of /etc/ora2pg/. Thanks to pawelbs for the report. - Add debug information before searching for custom type. - Attempt to fix error "ORA-01002: fetch out of sequence" when exporting data from a table with user defined types and ORACLE_COPIES. Thanks to pawelbs and Alex Ignatov fir the report. - Fix replacement of path to configuration file in scripts/ora2pg - Remove report sample from documentation about migration assessment report and replace it with a href link. Fix comment about export of domain index. - Always prefix table name with schema in Oracle row count, to prevent failure when the schema is not the connexion default. - Add pattern TOAD_PLAN_.* to the internal table exclusion list. - Fix modification of database owner search_path in import_all.sh auto generated script. Thanks to Stephane Tachoire for the report. 2016 02 29 - v17.1 This is a maintenance release to fix several issues reported in new TEST action. There is also some feature improvement: * Add OPTIONS (key 'true') on table FDW export when a column is detected as a primary key. * Add DELETE configuration directive that allow a similar feature than the WHERE clause to replace TRUNCATE call by a "DELETE FROM table WHERE condition". This feature can be useful with regular "updates". Thanks to Sebastien Albert for the feature request. Here is the complete list of other changes: - Fix the counter of user defined types and sequences in TEST action - Fix COPY import of data from column with user defined type with NULL value. - Fix DBD::Pg segmentation fault with direct INSERT import from column with user defined type. - Fix TEST action with multiple PG_SCHEMA export. Thanks to Michael Vitale for the report. - Fix documentation about PG_SCHEMA 2016 02 22 - v17.0 This new major release adds a new action type TEST to obtain a count of all objects at both sides, Oracle and PostgreSQL, to perform a diff between the two database and verify that everything have been well imported. It also fixes several issues reported by users. A new ora2pg command line option have been added to ora2pg script: * Add --count_rows command line option to perform a real row count on both side, Oracle and PostgreSQL, in TEST report. Here is the complete list of changes and bugfixes: - Prefix direct call to function with a call to PERFORM. Thanks to Michael Vitale for the feature request. - Fix revoke call on function with multiline parameters declaration. - Fix auto setting of internal schema variable with mysql. - Define ORACLE_HOME with the corresponding environment variable in generic configuration when available and --init_project is used. Thanks to Stephane Tachoires for the report. - Fix documentation about exporting view as table. - Remove some obsolete code and display information when a view is exported as table. - Fix empty LOB data export with Oracle Lob locator (NO_LOB_LOCATOR set to 0). - Fix data export of partitions with single process mode and when FILE_PER_TABLE is enabled. - Fix export of RAW data type. - Fix missing $ to call to self variable. Thanks to NTLIS and Sirko for the report. - Force FKey to be initially immediate when deferred is not set. Thanks to Stephane Tachoire for the report. - Fix count of check constraint when a schema is forced. - Allow TEST action on mysql database too with some improvements and bug fix on the feature. - Fix index column renaming in mysql export. - Fix dblink extraction query when an exclusion is set. - Fix sequence name auto generation for mysql serial number. - Add --count_rows command line option to make optional the real row count in TEST report. This is useful when you have lot of data and do not want to loose time in call to count(*). - Update documentation about the TEST action and usage, see chapter "Test the migration". - Apply schema context on PostgreSQL side with TEST action. - Add TEST action type to ask Ora2Pg to count rows and all objects at both sides, Oracle and PostgreSQL, to verify that everything have been well imported. - Fix missing export of foreign keys on multiple columns, ex: ALTER TABLE products ADD CONSTRAINT fk_supplier_comp FOREIGN KEY (supplier_id,supplier_name) REFERENCES supplier(supplier_id,supplier_name)... - Fix import of BLOB data using INSERT statements into the bytea. Thanks to rballer for the patch. - Fix missing export of FK when no schema is provided. 2016 01 13 - v16.2 This release fixes several issues, is more accurates on migration assessment report and adds some new ora2pg command line options: * Add --pg_dsn, --pg_user and --pg_pwd to be able to set the connection to import directly into PostgreSQL at command line. * Add -f option to script import_all.sh to force to not check user/database existing and skip their creation. Potential backward compatibility issues: * PG_SUPPORTS_CHECKOPTION is now enabled by default, you may want to migrate to PostgreSQL 9.4 or above. * Remove modification of CLIENT_ENCODING in generic configuration file with --init_project, use the default instead. * Remove modification of directive NLS_LANG to AMERICAN_AMERICA.UTF8 in generic configuration file with --init_project, use the default instead. Here is the complete list of other changes: - Adjust DBMS_OUTPUT calls to the migration assessment count. - Fix migration assessment count of call to cursor%ISOPEN and cursor%ROWCOUNT. - Replace zero date also with prepared statement with online PostgreSQL import and INSERT action. Thanks to Sebastian Albert for the report. - Remove REFERENCING clause in conditional triggers. Thanks to Raqua for the report. - Fix position of TG_OP condition when an exception is defined. Thanks to Raqua for the report. - Fix wrong replacement of SELECT with PERFORM when a comment was found between an open parenthesis and the select statement. Thanks to Raqua for the report. - Fix procedure return type with OUT and INOUT parameter. Thanks to Raqua for the report. - Fix rewrite of triggers with referencing clause. Thanks to Raqua for the report. - Fix default number of --human_days_limit in usage. - Fix replacement of placeholder %TEXTVALUE-d% to hide text string in query during function call rewrite. Thanks to Lorena Figueredo for the report. - Fix progress bar when a WHERE clause is used to limit the number of row to export. - Fix error "DBD::Pg::db do failed: SSL error: decryption failed or bad record mac" with pararellel table export (-P) and direct import to PostgreSQL via a ssl connection. Thanks to pbe-axelor for the report. - Fix missing index name in indexes creation. Thanks to Raqua for the report. - Fix pg DSN in import_all.sh autogenerated script. - Fix extraction of trigger. When the name of a column or something contained INSERTING, DELETING or UPDATING was converted to TG_OP = 'INSERT' or corresponding event. Thanks to Stanislaw Jankowski for the patch. - Fix multiple use of same column in check constraint and indexes of partitions when there was several schema with the same objects. - Fix default value for HUMAN_DAY_LIMIT to 5 when it is not defined in ora2pg.conf. - Fix double quote on column name in COPY export of partition tables Thanks to Chris Brick for the report. - Prevent case with several time same column in multicolumns unique constraints. Fix typo in previous patch. - Fix double quoted name with auto incremented sequence exported as serial. - Fix syntax error with MySQL data export with a WHERE clause using LIMIT. 2015 11 30 - v16.1 This release fixes several issues and adds some very useful features: * Generate automatically a new import_all.sh shell script when using option --init_project to help automate all import into PostgreSQL. See sh import_all.sh -? for more information. * Export Oracle bitmap index as PostgreSQL btree_gin index. This require the btree_gin extension and PostgreSQL >= 9.4. This is the default. * Auto set DEFINED_PK to the first column of a table that have a unique key defined that is a NUMBER. This allow data of any table with a numeric unique key to be extracted using multiple connexions to Oracle using -J option. Tables with no numeric unique key will be exported with a single process. * Improve BLOB export speed by using hex encoding instead of escape. This might speed up be BLOB export by 10. * Allow use of LOB locator to retrieve BLOB and CLOB data to prevent having to set LONGREADLEN. Now LONGREADLEN is set to 8KB. Old behavior using LONGREADLEN can still be enabled by setting NO_LOB_LOCATOR to 0, given for backward compatibility. Default is to use LOB locator. * Ora2Pg will also auto detect table with BLOB and automatically decrease DATA_LIMIT to a value lower or equal to 1000. This is to prevent OOM. * Improving indexes and constraints creation speed by using the LOAD action and a file containing SQL orders to perform. It is possible to dispatch those orders over multiple PostgreSQL connections. To be able to use this feature, PG_DSN, PG_USER and PG_PWD must be set. Then: ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 will dispatch indexes creation over 4 simultaneous PostgreSQL connections. This will considerably accelerate this part of the migration process with huge data size. * Domain indexes are now exported as b-tree but commented to let you know where possible FTS are required. * Add number of refresh ON COMMIT materialized view in detailed report. * Allow redefinition of numeric type, ex: NUMBER(3)::bigint to fix wrong original definition in Oracle. * Allow export of all schemas from an Oracle Instance when SCHEMA directive is empty and EXPORT_SCHEMA is enabled. All exported objects will be prefixed with the name of their original Oracle schema or search_path will be set to that schema name. Thanks to Magnus Hagander for the feature request. * Allow use of COPY FREEZE to export data when COPY_FREEZE is enabled. This will only works with export to file and when -J or ORACLE_COPIES is not set or default to 1. It can be used with direct import into PostgreSQL under the same condition but -j or JOBS must be unset or default to 1. Thanks to Magnus Hagander for the feature request. Some new configuration directives: * BITMAP_AS_GIN: enable it to use btree_gin extension to create bitmap like index with pg >= 9.4. You will need to create the extension by yourself: "create extension btree_gin;". Default is to create GIN index, when disabled, a btree index will be created. * NO_LOB_LOCATOR: to disable use of LOB locator and extract BLOB "inline" using a less or more high value in LONGREADLEN. * BLOB_LIMIT: to force the value of DATA_LIMIT for tables with BLOB. Default is to automatically set this limit using the following code: BLOB_LIMIT=DATA_LIMIT; while (BLOB_LIMIT > 1000) BLOB_LIMIT /= 10 * COPY_FREEZE: use it to use COPY FREEZE instead of simple COPY to speedup import into PostgreSQL. Here is the complete list of other changes: - Limite package function name rewrite to call with parenthesis after the function name to avoid rewriting names of other objects. - Fix extra replacement of function name with package prefix. On some condition it was done multiple time. - Set REPLACE_ZERO_DATE to -INFINITY in generic configuration when --mysql is enabled. - Fix extraction of partition with MySQL that was not limited to a single database. - Do some replacement on ORACLE_DNS and SCHEMA into generic configuration when --mysql is used for better understanding. - Add call to round() on -J parallelization when the auto detected column is a numeric with scale. - Add COMMIT to the difficulties migration assessment keywords as it need context analyzing. - Add call to cursor's %ROWCOUNT, %ISOPEN and %NOTFOUND to difficulties migration assessment keywords. - Replace call to CURSOR%ROWTYPE by RECORD. Thanks to Marc Cousin for the report. - Fix ALTER FUNCTION ... OWNER TO ... and REVOKE statement where functions parameters were missing. - Add Get_Env to the Oracle functions list for migration assessment. - Disable variable NO_LOB_LOCATOR and set LONGREADLEN to 8192 bytes to use LOB locators to extract BLOB in generic configuration file. - Fix call method "disconnect" on unblessed reference at line 9998. Thanks to Stephane Tachoires for the report. - Exclude from export objects name matching /.*\$JAVA\$.*/ and /^PROF\$.*/. - Fix migration assessment report when created during the package export. - Force writing Oracle package body in separate files when FILE_PER_FUNCTION is enabled and PLSQL_PGSQL disable to obtain package source code. - Fix case where sequence max value is lower than start value, in this case, set max value = start value. - Fix missing newline after each package file to import in global package.sql file when FILE_PER_FUNCTION is enabled. - Remove export of user PUBLIC in GRANT export. - Set DISABLE_TRIGGERS to 1 in generic configuration file auto generated when ora2pg option --init_project is used. - Remove call to quote_reserved_words() with index column when it we detect a function based index, too much false positive are rewritten with SQL code like CASE...WHEN. - Update export_schema.sh to remove .sql files when there is not such object leaving export directory empty. - Prevent creating TBSP_INDEXES_tablespace.sql when no tablespaces are found - Update documentation on WHERE clause on how to limit the number of tuples exported for Oracle and MySQL to test data import. - Fix unlisted spatial indexes in assessment report. - Fix double quote on index name with index renaming and reserved keyword. - Do not try to export tablespaces, privileges and audited queries as non DBA user when USER_GRANT is enabled. - Remove carriage return from list file. - Force SCHEMA to database name with MySQL migration. - Fix missing declaration of _extract_sequence_info(). Thank to Yannick DEVOS for the report. - Add documentation about COPY_FREEZE directive and add a note about export of all schema. - Remove systematic schema name appended to table name on KETTLE export, this must only be true when EXPORT_SCHEMA is enabled. - Fix TO_NUMBER() wrong replacement when a function is called as a parameter. - Fix non converted DECODE() when they was called in an XMLELEMENT function. - Suppress MDSYS.SDO_* from MDSYS call in migration assessment cost. - Remove use of DBMS_STANDARD called with raise_application_error function - Fix STRING type replacement - Recreate README as a text file, not a man page. - Reformat changelog to 80 characters. - Add -t | --test command line option to ora2pg_scanner to be able to test all connections defined into the CVS list file. 2015 10 15 - v16.0 This major release improve PL/SQL code replacement, fixes several bugs and adds some major new features: * Full migration of MySQL database, it just work like with Oracle database. * Full migration assessment report for MySQL database. * New script, ora2pg_scanner, to perform a migration assessment of all Oracle and MySQL instances on a network. * Add technical difficulty level in migration assessment. * Allow migration assessment on client queries extracted from AUDIT_TRAIL (oracle) or general_log table (mysql). * Ora2Pg has a "made in one night" brand new Web site (still need some work). See http://ora2pg.darold.net/ Example of technical difficulty level assessment output for the sakila database with some more difficulties: Total 83.90 cost migration units means approximatively 1 man-day(s). Migration level: B-5 Here are the explanation of the migration level code: Migration levels: A - Migration that might be run automatically B - Migration with code rewrite and a human-days cost up to 10 days C - Migration with code rewrite and a human-days cost above 10 days Technical levels: 1 = trivial: no stored functions and no triggers 2 = easy: no stored functions but with triggers, no manual rewriting 3 = simple: stored functions and/or triggers, no manual rewriting 4 = manual: no stored functions but with triggers or views with code rewriting 5 = difficult: stored functions and/or triggers with code rewriting This is to help you to find the database that can be migrated first with small efforts (A and B) and those who need to conduct a full migration project (C). This release has also some new useful features: * Export type SHOW_TABLE now shows additional information about table type (FOREIGN, EXTERNAL or PARTITIONED with the number of partition). * Connection's user and password can be passed through environment variables ORA2PG_USER and ORA2PG_PASSWD to avoid setting them at ora2pg command line. * Improve PL/SQL replacement on ADD_MONTH(), ADD_YEAR(), TRUNC(), INSTR() and remove the replacement limitation on DECODE(). * Add detection of migration difficulties in views, was previously reserved to functions, procedures, packages and triggers. * Replace values in auto generated configuration file from command line options -s, -n, -u and -p when --init_project is used. * Adjust lot of scores following new functionalities in Ora2Pg, ex: dblink or synomyms are now easy to migrate. There is some new command line options to ora2pg script: * -m | --mysql : to be used with --init_project and -i option to inform ora2pg that we work with a MySQL format * -T | --temp_dir : option to be able to set a distinct temporary directory to run ora2pg in parallel. * --audit_user : option to set the user used in audit filter and enable migration assessment report on queries from AUDIT_TRAIL (oracle) or general_log table (mysql). * --dump_as_sheet and --print_header options to be able to compute a CSV file with all migration assessment from a list of oracle database. * --dump_as_csv option to report assessments into a csv file. It will not include comments or details, just objects names, numbers and cost. Backward compatibility: - Change NULL_EQUAL_EMPTY to be disabled by default to force change in the application instead of transforming the PL/SQL. This release adds some new configuration directives: * MYSQL_PIPES_AS_CONCAT: Enable it if double pipe and double ampersand (|| and &&) should not be taken as equivalent to OR and AND. * MYSQL_INTERNAL_EXTRACT_FORMAT: Enable it if you want EXTRACT() replacement to use the internal format returned as an integer. * AUDIT_USER: Set the comma separated list of user name that must be used to filter from the DBA_AUDIT_TRAIL or general_log tables. * REPLACE_ZERO_DATE: "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL by default, use it to use the date of your choice. * INDEXES_RENAMING: force renaming of all indexes using tablename_columnsname Very useful for database that have multiple time the same index name or that use the same name than a table. * HUMAN_DAYS_LIMIT: default to 5 days, used to set the number of human-days limit for migration of type C. Here is the full list of other changes: - Remove list of acknowledgment that was not maintained anymore and some person may feel injured. Acknowledgment for patches or bug reports are always written to changelog, so this part reports to it now. - Fix bad trigger export when objects was enclosed in double quote and fix an additional bug in WHEN clause export. Thanks to Cyrille for the report. - Update documentation. - Update Makefile.PL with new script to install and new configuration directives in auto generated configuration file. - Update with new and missing files. - Add a Perl Module dedicated to MySQL database object discovery and export, lib/Ora2Pg/MySQL.pm. - Fix function based index type replacement in previous commit. - Do not report indexes with just DESC as function based index like Oracle report it. Thanks to Marc Cousin for the report. - Some excluded table was missing in the previous patch. - Remove use of DBI InactiveDestroy call when a fork is done and replace it to a single use AutoInactiveDestroy at connection. This require DBI>=1.614. - Add SDO_* tables and OLS_DIR_BUSINESSES in table exclusion list to fix issue #124 when no schema is provided. Thanks to Kenny Joseph for the report. - Fix partition prefix. - Remove UNIQUE keyword from spatial index. - Fix alter triggers function with missing parenthesis. Thanks to Spike Hodge MWEB for the report. - Fix export of foreign keys when they was defined in lowercase. Thanks to Spike for the report. - Fix wrong offset when rewriting ROWNUM with LIMIT+OFFSET. Thanks to Marc Cousin for the report. - Allow -INFINITY to be used to replace zero date. - Migration assessment in hour-day are now set to 1 man-day, we do not need such a precision and it is easier to process csv report. Thanks to Stephane Tachoire for the report. - Fix some issue with FDW and WKT spatial export. Add migration assessment of queries from the AUDIT_TRAIL table. - Adjust assessment units of some objects and add QUERY migration weight. - Rewrite information about migration levels. - Fix speedometer in progress bar, it will now shows the current speed in tuples/sec and the speed and time related to a table when export ended for the object. Thanks to Alex Ignatov for the report. - Fix break line when export data using INSERT mode. Thanks to Vu Bui for the report. - Do not display line about non existent objects in migration assessment reports. - Fix date default value for date when value is 0000-00-00 00:00:00 - Suppress display of title for function and trigger details when there is no details. - Remove INSTR() from the list of Oracle function that are not supported. It is now replaced by position(). - Fix condition to call _get_largest_tables(). - Fix some minor issues in OUT/INOUT type returned by a function. - Fix default value that may appears unquoted. - Fix several issues on partition export: column with function, index on default partition table and plsql to plpgsql translation in check condition. - Fix some minor issues. - Replace values from command line options -s, -n, -u and -p in --init_project auto generated configuration file. Thanks to Stephane Tachoire for the feature request. - Fix wrong object count in SHOW_REPORT. Thanks to Stephane Tachoire for the report. - Use DBA_SEGMENTS to find database size when USER_GRANT is disable, aka user is a DBA - Remove report of Migration Level when --estimate_cost is not enabled. - Add missing BINARY_INTEGER for type replacement. - Always exclude function squirrel_get_error_offset() that is created by the Squirrel Oracle plug-in. - Adjust assessment scores following new functionalities in Ora2Pg, ex: autonomous transaction, dblink or synomyms are now easy to migrate. - Remove man page from source, it is auto generated by Makefile.PL and make. - Fix unterminated DECODE replacement when there was more than 5 parameters to DECODE() and remove the limitation to 10 parameters. There is no more limit in the number of decode parameters. Thanks to Mael Rimbault for the report. - Remove inclusion of unwanted object when exporting a limited list of view with ALLOW. - Disable unsupported recursive query used to reorder views when Oracle version is 11gR1. Thanks to Mael Rimbault for the patch. - Add PLPGSQL replacement of INSTR() by POSITION(). Thanks to Mael Rimbault for the report. - Add difficulty level information in migration assessment, this include a new configuration directive HUMAN_DAYS_LIMIT (default to 5 days) to set the number of human-days limit for migration of type C. - Add MERGE with a migration cost of 3, still need work be replaced by ON CONFLICT. - Remove some redundant regular expressions. - Fix escaped commas not working properly in MODIFY_TYPE. A MODIFY_TYPE value like `TABLE1:COL4:decimal(9\,6)` was leading to a column like `col4 decimal(8#nosep#3)` in the SQL dump file that was generated. This fixes the output to be `col4 decimal(8,3)`. Thanks to Nick Muerdter for the patch. - Strip default "empty_clob()" values from table dumps. This function does not exist in Postgres and is not necessary. Thanks to Nick Muerdter for the patch. - Fix undesired double quoting of column name in function based indexes. - Fix issue with Perl < 5.8 "Modification of a read-only value attempted" - Fix retrieving of table size on Oracle 8i. - Add auto double quoting of object name with unauthorized characters. Thanks to Magnus Hagander for the feature request. - Automatically double quote object name beginning with a number - Fix missing DESC part in descending indexes. Thanks to Magnus Hagander for the report. - Fix case where a column name in oracle is just a number (e.g. the column is called "7"), it will be created in postgres without quoted identifier, which fails. Thanks to Magnus Hagander for the report. - Fix "reqs/sec" display in debug mode. Thanks to Laurent Martelli for the patch - Fix export if Oracle procedure is created without a parameter. Thanks to dirkgently007 for the report. - Fix CSV report output. - Fix triggers from file parser. - Add a test on triggers return to handle case where it is triggered on DELETE + other(s) event(s). In this case a test is done on the TG_OP to return OLD if event is DELETE or NEW in other case. Thanks to Dominique Legendre for the suggestion. - Change NULL_EQUAL_EMPTY to be disabled by default to force change of the application instead of transforming the PL/SQL. - Change score of SYNONYM and DBLINK in the migration assessment. - Add conversion of Oracle type STRING into varchar(n) or text. - Add information about libaio1 requirement for instant client - Remove extra space when calling ora2pg_get_efile() used to export BFILE into EFILE. Thanks to Dominique Legendre for the export. 2015 06 01 - v15.3 This is a maintenance release only that fixes several minor bugs and typos. The configuration file have been entirely rewritten to classify configuration directives in section for better understanding. Here is the full list of changes: - Ora2Pg will use DEFAULT_SRID when call to sdo_cs.map_oracle_srid_to_epsg() returns an ORA-01741 error. Mostly because there's no SRID defined for that column in ALL_SDO_GEOM_METADATA. The error message will still be displayed but a warning will explain the reason and ora2pg will continue with default value. Thanks to kazam for the report. - Add current setting for NLS_TIMESTAMP_FORMAT and NLS_DATE_FORMAT to the SHOW_ENCODING report. - Change default value for GEOMETRY_EXTRACT_TYPE to INTERNAL instead of WKT. - Change generic configuration file behavior with BINMODE parameter commented if it was previously uncommented. This will force to use the default value. - Fix potential issue with max open file limit with unclosed temporary file. Thanks to Marc Clement for the report. - Fix use of SECURITY DEFINER in SYNONYM export. - Fix parsing of editable function/procedure/package from input DML file. - Fix case where variable $2 and $3 was null after a too early call of a new substitution in read_view_from_file(). Thanks to Alex Ignatov for the patch. - Add support to "create or replace editionable|noneditionable" from input DML files. Thanks to Alex Ignatov for the report. - Fix unknown column HIGH_VALUE from *_TAB_PARTITIONS in Oracle 8i. Thanks to Sebastian Fischer for the patch. - Fix call to ALL_MVIEW_LOGS object which not exists with Oracle 8i. Thanks to Sebastian Fischer for the report. - Fix Error ORA-22905 while Get the dimension of the geometry by looking at number of element in the SDO_DIM_ARRAY. Thanks to jkutterer for the patch. - Remove reordering export of view for Oracle database before 11g. Thanks to kyiannis for the report. - Fix several some typos and a bunch of misspelled. Thanks to Euler Taveira for all the patches. - Fix missing Oracle database version before looking at function security definer. Thanks to kyiannis for the report. 2015 04 13 - v15.2 This new minor release fixes some issues and adds two new configuration directives: * ORA_INITIAL_COMMAND to be able to execute a custom command just after the connection to Oracle, for example to unlock a security policy. * INTERNAL_DATE_MAX to change the behavior of Ora2Pg with internal date found in user defined types. This version will also automatically re-order exported views taking into account interdependencies. Here is the full list of changes: - Add INTERNAL_DATE_MAX configuration directive with default to 49 to be used when reformatting internal date returned with a user defined type and a timestamp column. DBD::Oracle only return the internal date format 01-JAN-77 12.00.00.000000 AM so it is difficult to know if the year value must be added to 2000 or 1900. We takes the default behavior where date are between 1950 and 2049. - Remove extra CHAR and BYTE information from column type. Thanks to Magnus Hagander for the report. - Re-order views taking into account interdependencies. Thanks to Kuppusamy Ravindran and Ulrike for the suggestion and the Oracle query. - Fix case sensitivity in function based indexes. Thanks to Kuppusamy Ravindran for the report. - Fix PERFORM wrong replacement and infinite loop processing DECODE in some condition. Thanks to Didier Brugat for the report. - Fix replacement of boolean value in DEFAULT value at table creation. Thanks to baul87 for the report. - Add ORA_INITIAL_COMMAND configuration directive to be able to execute a custom command just after the connection to Oracle, to unlock a policy for example. Thanks to Didier BRUGAT for the feature request. - Fix alias in from clause when an XML type is found. Thanks to Lance Jacob for the record. - Invert condition on excluding temporary file with Windows OS. Thanks to kazam for the report. - Remove start time and global number of rows from _dump_table() parameters they are not used anymore. - Remove use of temporary file on Windows operating system. - Disable parallel table export when operating system is Windows. - Fix export of objects with case sensitivity using ALLOW or EXCLUDE directives. Thanks to Alexey Ignatov for the report. - Fix export of triggers from recycle bin. - Fix count of synonym in assessment report. - Add list of tables created by OEM to the exclusion list. - Fix look at default configuration file and set mode of export_schema.sh to executable by default. Thanks to Kuppusamy Ravindran for the report. - Add AUTHORIZATION to the list of PostgreSQL reserved word. Thanks to Kuppusamy Ravindran for the report. - Display a warning when an index has the same name as the table itself so that you can renamed it before export. Thanks to Kuppusamy Ravindran for the feature request. - Fix export of function based indexes with multiple column. Thanks to Kuppusamy Ravindran for the report. - Modify ora2pg script to return 0 on success, 1 on any fatal error and 2 when a child process die is detected. - Change the way the generic configuration file is handle during project initialization. You can use -c option to copy your own into the project directory. If the file has the .dist extension, ora2pg will apply the generic configuration on it. Thanks to Kuppusamy Ravindran for the report and features request. - Add debug information when cloning the Oracle connection. - Force return of OLD when the trigger is on DELETE event 2015 02 06 - v15.1 New minor release just to fix two annoying bugs in previous release. - Fix replacement of function name which include SELECT in their name by PERFORM. Thanks to Frederic Bamiere for the report. - Fix creation of sources subdirectories when initializing a new migration project. 2015 02 04 - v15.0 This major release improve PL/SQL code replacement, fixes several bugs and adds some new useful features: - Add support to the PostgreSQL external_file extension to mimic BFILE type from Oracle. See https://github.com/darold/external_file for more information. - Allow export of Oracle's DIRECTORY as external_file extension objects This will also try to export read/write privilege on those directories. - Allow export of Oracle's DATABASE LINK as Oracle foreign data wrapper server using oracle_fdw. - Allow function with PRAGMA AUTONOMOUS_TRANSACTION to be exported through a dblink wrapper to achieve the autonomous transaction. - Allow export of Oracle's SYNONYMS as views. Views can use foreign table to create "synonym" on object of a remote database. - Add trimming of data when DATA_TYPE is used to convert CHAR(n) Oracle column into varchar(n) or text. Default is to trim both side any space character. This behavior can be controlled using two new configuration directives TRIM_TYPE and TRIM_CHAR. - Add auto detection of geometry constraint type and dimensions through spatial index parameters. This avoid the overhead of sequential scan of the geometric column. - Add support to export Oracle sub partition and create sub partition for PostgreSQL with the corresponding trigger. - ALLOW and EXCLUDE directives are now able to apply filter on the object type. Backward compatibility can not be fully preserved, older definition will apply to current export type only, this could change your export in some conditions. See documentation update for more explanation. - Add PACKAGE_AS_SCHEMA directive to change default behavior that use a schema to emulate Oracle package function call. When disable, all calls to package_name.function_name() will be turn into package_name_function_name() just like a function call in current schema. - Add FKEY_OPTIONS to force foreign keys options. List of supported options are: ON DELETE|UPDATE CASCADE|RESTRICT|NO ACTION. - Add rewriting of internal functions in package body, those functions will be prefixed by the package name. Thanks to Dominique Legendre for the feature request. Some change can break backward compatibility and make configuration directives obsolete: - The ALLOW_PARTITION configuration directive has been removed. With new extended filters in ALLOW/EXCLUDE directive, this one is obsolete. Backward compatibility is preserved but may be removed in the future. - ALLOW and EXCLUDE directives do not works as previously. Backward compatibility may be preserved with some export type but may be broken in most of them. See documentation. - It is recommended now to leave the NLS_LANG and CLIENT_ENCODING commented to let Ora2Pg handle automatically the encoding. Those directives may be removed in the future. Here is the full changelog of the release: - Declares SYNONYM views as SECURITY DEFINER to be able to grant access to objects in other schema. - Fix wrong replacement of data type in function body. Thanks to Dominique Legendre for the report. - Fix missing column name replacement on trigger export when REPLACE_COLS is defined. Thanks to Dominique Legendre for the report. - Fix missing table replacement on trigger export when REPLACE_TABLES is defined. Thanks to Dominique Legendre for the report. - Fix case where IS NULL substitution was not working. Thanks to Dominique Legendre for the report. - Remove double exclusion clause when multiple export type is used with same column name and no values defined. - Allow parsing of DATABASE LINK and SYNONYM from a DDL file. - Add DIRECTORY export type to export all Oracle directories as entries for the external_file extension. This will also export read/write privilege on those directories. Thanks to Dominique Legendre for the feature request. - Review documentation about NULL_EQUAL_EMPTY. - Fix missing code to replace IS NULL as coalesce(...). Thanks to Dominique Legendre for the report. - Add external_file schema to search_path when BFILE is set to EFILE in directive DATA_TYPE. Thanks to Dominique Legendre for the request. - Remove IF EXIST clause to oracle function created by Ora2Pg for BFILE export. Thanks to Dominique Legendre for the report. - Add support to the PostgreSQL external_file extension to mimic BFILE type from Oracle. See https://github.com/darold/external_file for more information. - Add auto detection of geometry constraint type and dimensions through the spatial index parameters first. This avoid the overhead of sequential scan of the geometric column. - Remove lookup at package function when not required. - Fix issue with database < 10g that do not have the DROPPED column into the ALL_TABLES view. Thanks to Lance Jacob for the report. - Add trimming of data when DATA_TYPE is used to convert CHAR(n) Oracle column into varchar(n) or text column into PostgreSQL. Default is to trim both side any whitespace character. This behavior can be controlled using the new configuration directives TRIM_TYPE and TRIM_CHAR. - Update copyright year. - Add assessment cost for object TABLE SUBPARTITION and review cost for object DATABASE LINK. - Update documentation about SYNONYM export. - Allow export of SYNONYMS as views with a new export type: SYNONYM. - Fix object exclusion function with Oracle 8i and 9i. Thanks to Lance Jacob for the report. - Fix INTERVAL YEAR TO MONTH and DAY TO SECOND with precision. - Remove unused pragma from the cost assessment. - Suppress PRAGMA RESTRICT_REFERENCES, PRAGMA SERIALLY_REUSABLE and INLINE from the PLSQL code. There is no equivalent and no use in plpgsql. - Fix several issues in function/procedure/package extraction from file input and some other related bug. - Remove single slash and \\r from function code. - Remove schema from package name with input file to avoid creating function with SCHEMA.PKGNAME.FCTNAME - Fix ALLOW/EXCLUDE ignored with type COPY or INSERT. Thanks to thleblond for the patch. - Fix setting of NLS_NUMERIC_CHARACTERS and NLS_TIMESTAMP_FORMAT with multiprocess, the session parameters was lost with the cloning of the database handle. Thanks to thleblond for the patch. - Fix issue that could produce errors "invalid byte sequence" when dumping data to pg database by forcing the client_encoding when PG_DSN is set. Thanks to thleblond for the patch. - Fix issue to add parenthesis with function with no parameters and wrong use of PERFORM in cursor declaration. Thanks to hdeadman for the report. - Fix broken export of function or procedure without parameter in package body. Thanks to hdeadman for the report. - Fix ERROR: "stack depth limit exceeded" generated by an infinite loop in partition trigger when there is no default table when value is out of range. - Add support to Oracle sub partition export. - Fix issue with procedure in package without parameters. - Enable DISABLE_SEQUENCE in generic configuration file. - Fix unwanted alter sequence in data export when there is table allowed or excluded. - Fix initial default values of command line parameter that prevent value in configuration file to be taken. - Fix non working global definition of table in ALLOW and EXCLUDE directive with COPY and INSERT export. - Update ora2pg.spec, thanks to bbuechler for the patch. - Close temporary files before deleting them, on Windows if they are not explicitly closed there are not deleted. Thanks to spritchard for the patch. - Force schema name to be uppercase when PRESERVE_CASE is disable (default). Thanks to Jim Longwill for the report. - Add rewriting of internal functions in package body, those functions will be prefixed by the package name. Thanks to Dominique Legendre for the feature request. - Fix type replacement in user defined type. Thanks to Dominique Legendre for the report. - Add filter with INSTEAD OF triggers on views to TRIGGER export type. Thanks to Dominique Legendre for the feature request. - Fix replacement of function name when PACKAGE_AS_SCHEMA is disabled. - Fix PLSQL_PGSQL that was always set to 0 when -p was not used even if configuration directive PLSQL_PGSQL was activated. Thanks to Dominique Legendre for the report. - Remove ALTER SCHEMA ... OWNER TO ... when CREATE_SCHEMA is not enable. Thanks to Dominique Legendre for the report. - Add DBLINK export to be created as foreign data wrapper server. Thanks to the BRGM for the feature request. - Remove ALLOW_PARTITION configuration directive, with extended filter in ALLOW/EXCLUDE directive, this one is obsolete. Backward compatibility is preserved. - Add documentation about extended filters in ALLOW and EXCLUDE directive. - Update documentation about VIEW_AS_TABLE and remove statement change with export TYPE is VIEW. - Add filter to grant export on functions, sequences, views, etc. - Fix GRANT in ALLOW or EXCLUDE filters. - Add commented order: "REVOKE ALL ON FUNCTION ... FROM PUBLIC;" when the function is declared as SECURITY DEFINER. - Prevent collecting column information with SHOW_TABLE export type. - Fix default value SYSTIMESTAMP to CURRENT_TIMESTAMP, and remove DEFAULT empty_blob(). Thanks to hdeadman for the report. - ALLOW and EXCLUDE directives are now able to apply filter on the object type. Backward compatibility can not be fully preserved, older definition will apply to current export type only, this could change your export in some conditions. See documentation update for more explanation. Thanks to the BRGM for the feature request. - Force function to be created with SECURITY DEFINER when AUTHID in table ALL_PROCEDURES is set to DEFINER in Oracle. This only works with Oracle >= 10g. Thanks to Dominique Legendre for the feature request. - Add PACKAGE_AS_SCHEMA configuration directive to change default behavior to use a schema to emulate Oracle package function call. When disable all call to package_name.function_name() will be turn into package_name_function_name() just like a function call in current schema. Thanks to the BRGM for the feature request. - Add a note to documentation about the way to convert srid into Oracle database instead of in Ora2Pg. Thanks to Dominique Legendre for the hint. - Fix documentation about SHOW_ENCODING export type. - Remove use of REGEX_LIKE with Oracle version 9. Thanks to Lance Jacob for the report. - Replace new FKEY_OPTIONS by FKEY_ADD_UPDATE configuration directive with three possible values: always, never and delete. It will force or not Ora2Pg to add "ON UPDATE CASCADE" on foreign keys declaration. - Allow FORCE_OWNER to work with all exported objects. Thanks to BRGM for the feature request. - Add FKEY_OPTIONS to force foreign keys options. List of supported options are: ON DELETE|UPDATE CASCADE|RESTRICT|NO ACTION. Thanks to the BRGM for the feature request. - Fix ambiguous column in view extraction. Thanks to Dominique Legendre for the report. - Fix replacement of TYPE:LEN by boolean, ex: REPLACE_AS_BOOLEAN CHAR:1. Thanks to jwiechmann for the report. - Fix error ORA-00942 where Ora2Pg try to export data from a view defined in VIEW_AS_TABLE configuration directive. - Update list of excluded Oracle schema to the documentation. - Fix export of all views with comments when VIEW_AS_TABLE is set. - Fixed some typos in the generated sample configuration file. Thanks to Hal Deadman for the patch. - Limit column information export to the type of object extracted. - Remove call to MDSYS in SQL code. Thanks to Dominique Legendre for the report. - Add more Oracle schema to the exclusion list. - Fully remove join on DBA_SEGMENTS to retrieve the list of tables, views and comments. Replaced by ALL_OBJECTS. Thanks to Dominique Legendre for the help. - Exclude JAVA\$.* tables and fix tables list query to include newly created tables with no segments. Thanks to Dominique Legendre for the fix. - Fix regex that convert all x = NULL clauses to x IS NULL to not replace := NULL too. - Autodetect unusual characters in owner name when extracting data and used it embeded into double quote. - Replace single return with return new in trigger code. Thanks to Dominique Legendre for the report. 2014 11 12 - v14.1 This is a maintenance release only mainly to add patches that was not been applied in previous major release. - Remove ALLOW_CODE_BREAK, it is no more useful. - Change output of SHOW_ENCODING to reflect change to default encoding. - Comment ALLOW_PARTITION in default configuration file - Add QUERY and KETTLE export type in configuration file comments. 2014 11 05 - v14.0 This major release adds full export of Oracle Locator or Spatial geometries into PostGis, SDO_GEOM functions and SDO_OPERATOR are also translated. This export adds the following features: 1. Basic and complex geometry types support 2. Geometry data conversion from Oracle to PostGIS 3. Spatial Index conversion 4. Geometry metadata / constraints support 5. Spatial functions conversion For spatial data export, you have three choice, WKT to export data using SDO_UTIL.TO_WKTGEOMETRY(), WKB to export data using SDO_UTIL.TO_WKBGEOMETRY() and INTERNAL to export geometry using a Pure Perl library. Unlike the first two methods, INTERNAL is fast and do not raise Out Of Memory. The export is done in WKT format so that you can verify your geometry before importing to PostgreSQL. Other additional major features are: - Parallel table processing. - Auto generation of migration template with a complete project tree. - Allow user defined queries to extract data from Oracle. Parallel table processing is controlled by the -P or --parallel command line options or the PARALLEL_TABLE configuration directive to set the number of tables that will be processed in parallel for data extraction. The limit is the number of cores on your machine. Ora2Pg will the open one connection to Oracle database for each parallel table extraction. This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS, so the real number of process that will be used is (PARALLEL_TABLES * JOBS). The two options --project_base and --init_project when used indicate to Ora2Pg to create a project template with a work tree, a generic configuration file and a shell script to export all objects from the Oracle database. So that you just have to define the Oracle database connection into the configuration file and then execute the shell script called export_schema.sh to export your Oracle database into files. Here a sample of the command and the project's tree. ora2pg --project_base /tmp --init_project test_project /tmp/test_project/ config/ ora2pg.conf data/ export_schema.sh reports/ schema/ fdws/ functions/ grants/ kettles/ mviews/ packages/ partitions/ procedures/ sequences/ tables/ tablespaces/ triggers/ types/ views/ sources/ functions/ mviews/ packages/ partitions/ procedures/ triggers/ types/ views/ It create a generic config file where you just have to define the Oracle database connection and a shell script called export_schema.sh. The sources/ directory will contains the Oracle code, the schema/ will contains the code ported to PostgreSQL. The reports/ directory will contains the html reports with the migration cost assessment. Sometime you may want to extract data from an Oracle table but you need a custom query for that. Not just a "SELECT * FROM table" like Ora2Pg do but a more complex query. The new directive REPLACE_QUERY allow you to overwrite the query used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. If you have multiple table to extract by replacing the Ora2Pg query, you can define multiple REPLACE_QUERY lines. For example: REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] Other new features are: - Export of declaration of language C function. Previous version was not exporting function with no code body like external C function. - Export of COMMENT from views. - Function to replace some call to SYS_CONTECT(USERENV, ...) by the PostgreSQL equivalent. - Add POSTGIS_SCHEMA configuration directive to add the dedicated PostGis schema into the search_path. - Add PG_SUPPORTS_IFEXISTS configuration directive to be able to suppress IF EXISTS call in DDL statement generated by Ora2Pg. - Triggers are now all excluded/allowed following the table names specified in the ALLOW and EXCLUDED directives - Allow automatic export of nested tables (TYPE+TABLE+COPY). One change is not fully backward compatible: Ora2Pg now use UTF8 by default on both side. On Oracle connection NLS_LANG is set to AMERICAN_AMERICA.AL32UTF8, NLS_NCHAR to AL32UTF8. On PostgreSQL side CLIENT_ENCODING to UTF8. For export that dump to files, Perl binmode is set to utf8. You can always change those default setting in configuration file, but it is not recommanded. Here is the full changelog of the release: - Fix inline comments into function declaration. Thanks to Marcel Huber for the report. - Fix case where SELECT ... INTO was wrongly replaced by PERFORM. - Fix DECODE() translation. Thanks to Dominique Legendre for the report. - Add replacement of SDO_OPERATOR into PostGis relationships. - Add replacement of SDO_GEOM spatial function to postgis ST_* functions. - Add GEOMETRY_EXTRACT_TYPE configuration directive to specify the geometry extracting mode: WKT (default), WKB and INTERNAL. - Add a pure Perl library to export SDO_GEOMETRY as a WKT representation. This is controlled by a new extraction type INTERNAL to use with the GEOMETRY_EXTRACT_TYPE configuration directive. - Remove USE_SC40_PACKAGE directive and any reference to this library, it is not useful now that we have the INTERNAL geometry extraction mode. - Fix replacement of varchar2 in PL/SQL function. - Fix bug in type replacement when default values used function. - Add export of declaration of language C function. Previous version was not exporting function with no code body like external function. - Fix create statement in export of view as table. Thanks to ntlis for the report. - Fix replacement of to_number without format. - Add export of COMMENT from VIEWS. - Add function to replace some call to SYS_CONTECT(USERENV, ...) by the PostgreSQL equivalent. - Fix parsing from file of tablespace. - Fix wrong alias name in FROM clause when extracting XML data. Thanks to Marc Sitges for the report. - Fix export of comments in FDW export, might be COMMENT ON FOREIGN TABLE. Thanks to David Fetter for the report. - Fix broken export of function based indexes. Thanks to Floyd Brown for the report. - Fix sequence with negative minvalue/maxvalue and negative increment. Thanks to jwiechmann for the report. - Fix forced owner to schema to the value of FORCE_OWNER when it is set to a user name. - Fix create schema when FORCE_OWNER is enabled. Thanks to Dominique Legendre for the report. - Add POSTGIS_SCHEMA configuration directive to add a schema to the search_path. Thanks to Dominique Legendre for the feature request. - Returns NULL when a geometry is NULL instead of calling ST_AsText with a null value. Thanks to Dominique Legendre for the report. - Add more explanation about values of CONVERT_SID. - Fix issue in DBMS_OUTPUT replacement. - Fix exclusion of default objects from type export. - When CONVERT_SRID is > 1 this value will be used to force the SRID value on all export. - Disable NULL_EQUAL_EMPTY in generic configuration when generating a project tree. - Add LOGMNR$ and RECAP$ in the exclusion objects list. - Fix performance issue in extracting data from geometry column and add AUDSYS,DVSYS and DVF to the list of schema to exclude. - Prefix table name with schema name on queries for retrieving data to avoid errors in multi schema export. - Add SDO_* cost to migration report. - Fix real number of Synonym that should be review. - Fix wrong report of CTXSYS synonym. - Enabled AUTODETECT_SPATIAL_TYPE by default. - Remove KETTLE and FDW export from the auto generated project. - Force the copy of /etc/ora2pg/ora2pg.conf.dist into the project directory with no more look at the current ora2pg.conf. Force autodetection of spatial type in the generic configuration. - Huge performance gain on querying information about Spatial column. Thanks to Dominique Legendre for the great help. - Fix wrong use of table alias with SEGMENT_NAME. - Add unified audit table (CLI_SWP$.*) from the exclusion list. - Fix operator in check condition of range partitions. Thanks to Kaissa Chellouche for the report. - Add to the internal exclusion list tables generated by spatial indexes MDRT_.*, sequences MDRS_.* and interMedia Text index DR$.*. Thanks to Dominique Legendre for the report. - Make REPLACE_TABLES and REPLACE_COLS work with VIEW. The view name and the columns aliases will be replaced. Take care that the table name or columns names in the statement will be kept untouched and need manual rewriting. Thanks to Sven Medin for the feature request. - Add PG_SUPPORTS_IFEXISTS configuration directive to be able to suppress IF EXISTS call in DDL statement generated by Ora2Pg. PostgreSQL below 9.x do not support this keywords. Thanks to George Kowalski fot the feature request. - Fix wrong substitution in EXECUTE ... USING statement, where parameters number was not prefixed by a $ sign. Thanks to Dominique Legendre for the report. - Fix document about KEEP_PKEY_NAMES that also affect unique key and not only primary key as it was specified in the documentation. Thanks to Dominique Legendre for the report. - Add tables generated by statistics on spatial index (MDXT_.*) into the internal exclusion list. This join the already excluded table generated by partition logging (USLOG$_.*) and materialized view logs (MLOG$_.*, RUPD$_.*) - Add DEFAULT_SRID configuration direction to permit change of the internal default EPSG srid 4326. - Fix new line after search_path settings. Thanks to Dominique Legendre for the report. - Triggers are now all excluded/allowed following the table names specified in the ALLOW and EXCLUDED directive, no more on there own name which had little interest. Thanks to Dominique Legendre for the feature request. - Add support to COPY export with Spatial objects. Thanks to Legendre Dominique for the great help to solve this problem. - Fix default SRID value when a NULL value is returned by Oracle, SRID 8307 and the corresponding EPSG SRID 4326. - Update documentation on relation between PARALLEL_TABLES and FILE_PER_TABLE - Add the -P or --parallel command line options and update documentation about parallel table processing. - Add PARALLEL_TABLES configuration directive to force ora2Pg to use on process and one connection per table up to the number of CPU specified. Thanks to menardorama for the feature request. - Add PARALLEL_TABLES configuration directive to force ora2Pg to use on process and one connection per table up to the number of CPU specified. Thanks to menardorama for the feature request. - Add --init_project and --project_base command line options to create a migration template with a complete project tree, a generic configuration file and script to automate export of all object in the project tree. - Fix unwanted space before AND returned by limit_to_tables(). Thanks to Alex Wang for the report. - Add note about regex inclusion/exclusion not working with 8i database in documentation - Fix regex inclusion/exlusion of table that was not more working since the inclusion of limit_to_tables() function. Thanks to alex wang for the patch - Exclude dropped tables (those who are in the recycle bin) from export. - When USER_GRANTS is disabled, aka login as dba user, force table list to be checked against DBA_SEGMENTS with SEGMENT_TYPE of type table or table partition. This could help solving some incomprehensible object found in Oracle view ALL_TABLES. - Fix query to retrieved list of tables, owner selection was set two time. - Add support to automatic nested table export (TYPE+TABLE+COPY). - Fix wrong export of materialized view log table. Thanks to Ronson Blossom for the report. - Update the SYSUSER array to exclude objects owned par those more users. - Fix unwanted export of overflow table of an index-organized table. Thanks to Ronson Blossom for the report. - Update the SYSUSER array to exclude objects owned par those users. - Display table owner in debug mode for SHOW_TABLE or SHOW_COLUMN. - Add a section to give hint about converting Oracle outer join syntax to ANSI. Thanks to Sven Medin for the links. - Fix issue #82 again. Thanks to Sven Medin fro the report. - Add first support to user defined queries to extract data from Oracle. This feature add a new configuration directive named REPLACE_QUERY. - Change program title when dump to file. - Fix MODIFY_TYPE directive that was broken when using type with space character. Thanks to Dmitry K. for the patch. - Show missing view name in debug mode when exporting some views as table. - Rewrite replace(a,b) with three arguments replace(a,b,'') for PostgreSQL. Thanks to Dominique Legendre for the report. - Convert all x <> NULL or x != NULL clauses to x IS NOT NULL. All x = NULL are converted into x IS NULL. Thanks to Dominique Legendre for the report. - Add warning at exit to signal when a OOM occurs. In that case, when a child Ora2Pg process was silently killed by the OOM killer there was no information that a failure occurs. 2014 06 02 - v13.0 This major release adds first support to export Oracle Spatial Objects to PostGis Spatial objects. There's also a new configuration directive to allow logging of statement failures to prevent Ora2Pg to abort and continue to load valid data. The other main feature is the possibility to convert DDL files without needing an Oracle database connection, until now this was reserved to files containing stored procedures. There's also several bug fixes. - Allow error logging during data import. This feature controlled by the LOG_ON_ERROR directive allow you to not abort the data import process when an error is encountered and to log to a file the COPY or INSERT statement that generate the error. After fixing the statement you will be able to load the missing data. Thanks to menardoram for the feature request. - Force export type to be INSERT when COPY is used and a table have a GEOMETRY column. I can not find a solution to export as copy statement for the moment. Thanks to Dominique Legendre and Vincent Picavet for the help. - Fix export of user defined type as object. Thanks to Shanshan Wang for the report. - Limit look up of objects to the ALLOW or EXCLUDE filter into the SQL query instead of the Perl code to avoid retrieving huge list of objects on such database. Thanks to menardorama for the feature request. - Add support to spatial data export in INSERT mode. Still need some work in COPY export mode if possible. - Fix query to retrieve SRID that broken with patch on CONVERT_SRID. - Fix wrong filter with ALLOW directive when getting list of partition. - Add GRANT export read from an input file. - Fix data type conversion when using input file and data type such varchar2(10 BYTE). - Add export of comment with TABLE and VIEW exports using an input file. - Add extraction of TABLESPACE from an input file. - Add support to SEQUENCE extraction from input file. - Fix wrong filter with ALLOW directive when exporting partition. The filter was done on partition name instead of table name, that mean that setting ALLOW directive was resulting in no export at all. Thanks to menardorama for the report. - Add CONVERT_SRID configuration directive to control the automatic conversion of SRID to standard EPSG using the Oracle SDO function sdo_cs.map_oracle_srid_to_epsg() Oracle function. Thanks to Dominique Legendre for the help. - Fix a typo in the create index prefix on partitioned tables. Thanks to menardorama for the patch. - Fix non replacement of destination during SHOW_COLUMN and COPY export. Using MODIFY_TYPE was only working in TABLE export. - Force pl/sql conversion with TABLE export to replace advanced default values. Fix code TRUNC(SYSDATE, MONTH) in default value and everywhere that should be: date_trunc(month,LOCALTIMESTAMP). Thanks to menardorama for the report. - Fix code regarding unique partition index naming. Thanks to menardorama for the report. - Add PREFIX_PARTITION configuration directive. When enabled it will force renaming all partition table name with the name of the parent table. Thanks to menardoram for the feature request. - Add AUTODETECT_SPATIAL_TYPE in configuration file and documentation about this new directive. - Add export of SDO_GEOMETRY column type. They are basically exported to the non-constrained "geometry" type with SRID if defined. When the configuration directive AUTODETECT_SPATIAL_TYPE is enable, Ora2Pg will try to autodetect the geometry type, the dimension and the SRID used to set a constrained geometry type. For example, in the first case column shape with Oracle type SDO_GEOMETRY will be converted as: shape geometry(GEOMETRY) or shape geometry(GEOMETRY, 4326) and in the second case, with constrained geometry type: shape geometry(POLIGONZ, 4326) with a three dimensional polygon. Thanks to Vincent Picavet for the feature request and specification. - Add support to spatial index read from file. - Add export of Oracle spatial index. For example, index: CREATE INDEX cola_spatial_idx ON cola_markets(shape) INDEXTYPE IS MDSYS.SPATIAL_INDEX; will be exported as CREATE INDEX cola_spatial_idx ON cola_markets USING GIST(shape); Thanks to Vincent Picavet / Oslandia for the feature request and explanations. - Allow TRIGGER export to parse an input file with Oracle DML orders. - Add PG_SUPPORTS_CHECKOPTION configuration directive to not remove WITH CHECK OPTION in create view statement. It is supported in PostgreSQL 9.4. - Allow VIEW export to parse an input file with Oracle DML orders. - Allow TABLE export to parse an input file with Oracle DML orders. - Add SYNCHRONOUS_COMMIT configuration directive disabled by default. This is the current behavior of Ora2Pg, it set synchronous_commit to off before data import to PostgreSQL. This is only used when you load data directly to PostgreSQL, the default is off to disable synchronous commit to gain speed at writing data. Some modified or old version of PostgreSQL, like Greenplum, do not have this setting. - Add some useful information for Windows user in documentation. Thanks to Roger Park for the report. - Fix case when parentheses are omitted in index creation. Thanks to Yuri Ushakov for the report. - Fix export type PACKAGE when ALLOW is defined to extract only some packages. Thanks to Maciej Bak for the report. - Fix INSERT export where backslash should be escaped and single be doubled in standard conforming string notation. Thanks to Yuri Ushakov for the report. - Add important note about LONGREADLEN and DATA_LIMIT that could need to be adjusted to avoid out of memory. Thanks to Mike Kienenberger for the patch. - Fix case sensitivity issue with export of comment on column. Thanks to Pierre Crumeyrolle for the report. - Fix export of RAW data in COPY mode, was missing a backslash. Thanks to jwiechmann for the report. - Fix RAW data export in COPY and INSERT mode, RAW data type is returned in hex by DBD::Oracle. Thanks to jwiechmann for the report. - Fix one release 8i condition. - Fix inexistent column USE_NO_INDEX with Oracle 8i and MVIEW export. - Enclose call to utf8::encode and utf8::valid into eval. - Fix export of constraint with Oracle 8i release. - Fix unrecognized fatal error with 8i database. Thanks to UnvorherSeba for the patch. - Revert change level of error from fatal to error, when querying materialized view. - Change level of error from fatal to error, when querying materialized view. 2014 01 28 - v12.1 This is a maintenance release with some minor bug fixes and a new configuration directive, INDEXES_SUFFIX, to allow appending a suffix to indexes names. - Fix example given for the WHERE configuration directive. Thanks to Bob Treumann for the report. - Add INDEXES_SUFFIX configuration option to allow append a suffix to indexes names. - Replace special charater ^M by \r as they are not supported by git. - Fix IF EXISTS in alter table of sub _drop_foreign_keys. Thanks to Francis Corriveau for the patch. - Fix isolation level when exporting data. Thanks to Ludovic Penet for the report. - Fix regression when ora2pg tries to create foreign keys on tables or to tables that are not selected for export. Thanks to Ludovic Penet. - Add information about backslashed comma into directive MODIFY_TYPE into Makefile.PL. - Add missing MODIFY_TYPE definition in documentation. - Allow backslashed comma into MODIFY_TYPE type redefinition. Example: TABLE1:COL3:decimal(9\,6),TABLE1:COL4:decimal(9\,6). Thanks to Mike Kienenberger for the report - Fix missing single cote into create_materialized_view() call. Thanks to Jacky Rigoreau for the patch. - Fix some typo in documentation, thanks to Mike Kienenberger for the report. - Add a chapter about installing DBD::Oracle into documentation. Thanks to Raghavendra for the patch. - Fix case sensitivity on external table name with FDW export type. Thanks to Guillaume Lelarge for the report. - Fix export of materialized views when PG_SUPPORTS_MVIEW is disabled. Thanks to Christian Bjornbak for the report. - Update copyright. 2013 10 22 - v12.0 This release fixes lot of issues and three new features. Using REORDERING_COLUMNS directive you will be able to reorder columns to minimized the footprint on disc, so that more rows fit on a data page. The PG_SUPPORTS_MVIEW will allow you to export materialized with native PostgreSQL 9.3 syntaxe. The USE_TABLESPACE variable will allow you to export object using their original tablespace. - Skip constraints on system internal columns (sys_nc...$) from export. - Fix missing output directory in generic psql file for data loading. - Add missing progress bar during TYPE and PARTITION export type. - Remove duplicated message in debug mode during Oracle reconnection. - Allow file input with create type declaration to use ora2pg converter. Unsupported syntax is signaled into the output file. - Exclude MLOG$.* and RUPD$.* table from export. - Prevent export of indexes and constraints during FDW export type. - Fix wrong total number of sequences shown in progress bar. - Remove warning when PG_DSN is define during a export type that do not support direct import into PostgreSQL. - Auto switch prefix from DBA to ALL when error 942 is returned when looking at tables informations. A hint is also displayed to ask for activating USER_GRANTS or connect using a user with DBA privilege. - Add REORDERING_COLUMNS configuration directive to allow reordering columns during the TABLE export. This could help to minimized the footprint on disc, so that more rows fit on a data page. Thanks to Christian Bjornbak for the feature request. - Fix call to unblessed reference at disconnect when direct import to pg is not used. Thanks to Christian Bjornbak for the report. - Fix regression in drop/create foreign keys and index during data export. Thanks to Christian Bjornbak for the report. - Fix truncate table error with parallel and direct data copy. Thanks to keymaper for the report. - Fix several other issues with parallel and direct data import. - Fix trigger export on multi files when FILE_PER_FUNCTION is enabled. - Fix issue on converting boolean values with non default values. Thanks to Christian Bjornbak for the report. - Fix boolean value for disabled key in default %BOOLEAN_MAP key/value. - Fix case where INTO was wrongly replaced by INTO STRICT. Thanks to Jacky Rigoreau for the report. - Fix case where label after a END was not removed. Thanks to Jacky Rigoreau for the report. - Fix discard of input file parsing. Fix PERFORM replacement in PL/SQL code wirh cursor. Thanks to Jacky Rigoreau for the report. - Enable PG_SUPPORTS_MVIEW by default and update documentation. - Replace DBA_DATA_FILES by USER_SEGMENTS to get database size to avoid error ORA-00942. Thanks to Pierre Boizot for the report. - Fix trigger conversion error. Thanks to Pierre Boizot for the report. - Add support to PostgreSQL 9.3 materialized view syntaxe, this need a new configuration directive PG_SUPPORTS_MVIEW to be enabled. - Update default configuration file and documentation about USE_TABLESPACE. - Add USE_TABLESPACE configuration directive to force ora2pg to use Oracle tablespace name with table, constraints indexes and indexes if tablespace in not in the default (TEMP, USERS, SYSTEM). Thanks to Rob Moolhuijsen for the feature request. - Allow DEFER_FKEY, when enabled during TABLE export, to create all foreign keys as DEFERRABLE and INITIALLY DEFERRED. Thanks to David Greco for the patch. - Fix non working ON_ERROR_STOP set to 0 during data export. - Lot of code changes to fix dump to file in multiprocess mode. Ora2Pg will also only drop/create constraints and indexes related to the allow/exclude tables, thanks to Maciej Bak for the report. - Force decimal character from Oracle output to be a dot. Thanks to Maciej Bak for the report. - Add default exclusion of Oracle recycle bin objects with name begining by BIN$. - Fix escaping quote in table and column comments. Thanks to realyota for the report. - Reduce DECODE migration cost from 2 to 1 unit. - Reduce OUTER JOIN (+) migration cost from 3 to 1 unit. - Add Time::HiRes to the requirement chapter for Perl <= 5.8. Thanks to Mike Kienenberger for the report. - Replace wrong use of --config instead of --conf into the documentation. Thanks to Mike Kienenberger for the report. - Fix regex used to rewrite CREATE VIEW code. Thanks to David Greco for the patch. - Fix an issue with oracle copies when primary key was negative. Thanks to David Greco for the patch. - Fix case sensitivity with SEQUENCE when preserve_case is enabled. Thanks to Jean-Max Reymond for the report. - Fix table COMMENT export when preserve_case is enabled. Thanks to Jean-Max Reymond for the report. 2013 05 28 - v11.4 This release fixes others several major issues on migration cost assessment that was not addressed in previous release, please upgrade. - Fix other major issues in migration cost assessment. - Redefine some migration cost values to be more precise. 2013 05 27 - v11.3 This release fixes several major issues on migration cost assessment, especialy with stored procedures with lot of lines or if you have lot of comments in that code. You may want to run your database evaluation again as the estimated times can be up to tree time lower on huge PL/SQL code. - Add full details about PL/SQL evaluation by ora2pg when --estimate_cost or ESTIMATE_COST is enable. This will display cost units per keywords detected in the function/package code. - Fix wrong cost unit assessment on PL/SQL code size, this bug generated very high migration cost assessment for functions/packages with lot of lines. Please run your tests again, estimated times can be up to tree time lower on huge code. - Remove comments before code evalution. - Fix file input parser for PL/SQL packages export when IS or AS was in the next line than the CREATE PACKAGE BODY ... - Exclude NOT NULL constraint from the count of CHECK constraints into the TABLE report. - Fix decimal precision in table migration assessment cost. - Fix typo in changelog. 2013 05 01 - v11.2 This release fixes several major issues especially with direct import of data into PostgreSQL and Windows port that was both broken. - Update doc about Windows multiprocess issues and acknowledgements. - Fix Windows OS issues using multiprocessing options by disabling multiprocess support on this plateform. When -J or -j will be used a warning will be displayed and Ora2Pg will simply run single process like in previous 10.x versions. Thanks to Jean Marc Yao Adingra for the report. - Fix RAW and LONG RAW export to ByteA. Thanks to Prabhat Tripathi for the report and testing. - Fix patch regression on multiple TRUNCATE call for a single table. Thanks to David Greco for the report. - Placed calls to DB handle InactiveDestroy outside the forked process to prevent fatal errors on Windows. Thanks to Jean Marc Adingra for the report. - Forked running processes are renamed into more readable name like "ora2pg logger" for the progress bar, "ora2pg - querying Oracle" when used with -J option and "ora2pg - sending to PostgreSQL" to better know what is the current job of the process. - Removed the use of /Y flag in Windows install script, this was causing error "dmake: Error code 130, while making install_all". Thanks to Jean-Marc Adingra for the report. - Fix direct import to PostgreSQL that was just producing nothing. Thank to David Greco for the patch. - Fix ora2pg usage documentation. - Add an underscore to CLIENT ENCODING in SHOW_ENCODING output to be the same as the configuration directive. UPGRADE: please reinstall all as most of the files have changed. 2013 04 07 - v11.1 This release adds partition data speed improvement by exporting data directly from and into the destination partitioned table. There's also some bug fix on RAW or LONG RAW data export and PL/SQL to PL/PGSQL code rewrite. - Adjust cost assessment for indexes, tables and tables partition. - Add comment to report of index partition about local index only. - Fix position of TRUNCATE TABLE in output file. - Fix export of data from RAW or LONG RAW columns, they was exported as hex string. Now data are converted using utl_raw.cast_to_varchar2() function before being escaped for insert into a bytea. Thanks to Alex Delianis for the report. - Fix issue with Oracle TIMESTAMP(0) data export that add a single ending point, ex: "2008-08-09 00:00:00.", this ending character is now removed by format_data_type(). Thanks to Pierre-Marie Petit for the report. - Fix typo on MODIFY_STRUCT description. - Force DEBUG to off in default configuration file. - Change range PARTITION operators in the check conditions, >= and < replaced by > and <=, corresponding to Oracle VALUES LESS THAN. - Add ALLOW_PARTITION to limit data export to a list of partition name. - PLSQL: Fix wrong replacement of SELECT by PERFORM during VIEW export. - Partitioned tables data is now imported directly into the destination tables instead of inserted into the main table and dispatched by the trigger. Ora2Pg will automatically detect the in/out table partition, there's nothing to configure. - PL/SQL: Do not allow decode() rewrite by case/when/else when there is a function call in it. - Fix Error when Compress::Zlib is not installed, this module is not mandatory. UPGRADE: please reinstall all as all files have changed. 2013 03 24 - v11.0 This is a new major release because it adds support to multiprocessing to export data in parallel mode, this allow to improve speed during data import by more than ten times. This multiprocessiing capabilities allow Ora2Pg to be closer than the speed of any ETL. To compare speed or allow using Kettle for data import, there's now a new export type to obtain Kettle XML transformation files. This release adds also lot of work on speed improvement to scan Oracle database with huge number of object. - Add documentation about JOBS, ORACLE_COPIES, DEFINED_PK configuration directive and informations about KETTLE export type. - Add KETTLE export type to generate XML transformation file definition for Penthatlo Data Integrator (Kettle). Thanks to Marc Cousin for the work. Example of use: ora2pg -c ora2pg.conf -t KETTLE -j 12 -J 4 -o loaddata.sh - Fix major bug in export of auto generated named constraint. Thanks to mrojasaquino fot the report. - Show number of rows in the top largest tables. - Add TOP_MAX description to the documentation. - Add the TOP_MAX directive to default configuration file and update documentation. Directive used to control the top N tables to show. - Add top N of largest tables in SHOW_TABLE, SHOW_COLUMN and SHOW_REPORT export type. - Fix progressbar output when ora2pg is interrupted by ctrl+c. - Add JOBS, ORACLE_COPIES and DEFINED_PK directives to configuration file. JOBS replacing THREAD_COUNT but backward compatibility is preserve. - Add 3 new command line options, -j | --jobs and -J | --copies, used to set the number of connection to PostgreSQL and Oracle for parallel processing. The third, -L | --limit is used to change DATA_LIMIT at command line. - Add multiprocess support on data export. With the help of Thomas Ogrisegg. - Add more schema in SYSUSERS that should not be exported. - Add full detailed information about SYNONYM in SHOW_REPORT. - Add MODIFY_TYPE configuration directive to allow some table/column type to be changed on PostgreSQL side during the export. - Fix objects type count in progressbar of SHOW_REPORT. - Restrict table and index in SHOW_REPORT to the tables defined in ALLOW and EXCLUDE directives. - Show total number of rows in SHOW_TABLE and SHOW_REPORT output. - Add top 10 of tables sorted by number of rows in SHOW_TABLE and SHOW_REPORT output. - Fix typo in SYNONYM objects. - Add report of top ten tables ordered y number of rows. - Rewrite most of the Oracle schema storage information extraction for speed improvement. - Use Hash to store column informations. - Fix %unique_keys declaration in _table() method. - Remove call to _table_info() from SHOW_REPORT code as those informations are already loaded with the _table() method. - Fix missing column definition on TABLE export. - Add progress bar during output generation following export type. - Add STOP_ON_ERROR configuration directive to enable/disable the call to ON_ERROR_STOP into generated SQL scripts. Thanks to Ludovic Penet for the feature request. - Huge speed improvement on columns informations retrieving. - Fix progress bar to keep the total number of tables related to the ALLOW or EXCLUDE configuration directives. Thanks to Ludovic Penet for the report. - Change return type of function _table_info(), it now returns data instead of the database handle. - Improve speed on indexes and constraints extraction for database with huge number of tables. - Improve performance to retrieve columns information and comments. - Remove report of column details during export in debug mode, use SHOW_COLUMN instead. - Remove call to upper() in objects owner condition to improve performance with database with huge number of objects. - Add a fix to not export foreign key for exclude tables. Thanks to Ludovic Penet for the report. - Fix Windows install issue with copying ora2pg.conf.dist. Thanks to Dominique Fourdrinoy for the report. - Increase the cost of Oracle function not converted to PG automatically. UPGRADE: reinstall all is required to override the old installation, you may use the new ora2pg.conf.dist file which included the new configuration directives. 2013 01 15 - v10.1 This release adds HTML report for migration cost assessment and some bug fix. - Fix global where should not be overwritten. Thanks to Dan Harbin for the patch. - Fix bug/typo in boolean replacement, where a colon was used instead of a single quote. Thanks to Alex Delianis for the patch. - Update copyright. - Add detection of additional Oracle functions for better migration cost assessment. - Update documentation. - Force report detail in lowercase. - Added information about the migration cost value to the reports. - Add --dump_as_html command line option and DUMP_AS_HTML configuration directive. - Allow migration report to be generated as HTML. - Separate report generation code from data collection code. 2012 12 12 - v10.0 This is the first version of Ora2Pg 10.x series, that is a major release. Overall numerous improvements and bugs fixes there's now a new export type: SHOW_REPORT that will output a report of all objects contained in your Oracle database and some comments on how they will be exported. With this report you can use a new directive ESTIMATE_COST to ask to Ora2Pg to evaluate the database migration cost in terms of man days. There's also an other new configuration directive EXTERNAL_TO_FDW, disable by default, to permit the export of all Oracle external tables as file_fdw foreign tables. The database content report and the migration cost estimation is a work in progress so all feedback on these new features are welcome. Here is the complete changelog: - Update documentation about ora2pg usage and new feature. - Fix quote escaping on table comments. Thanks to Sebastian Fischer. - Fix some other issues with 8i databases, added database version auto- detection to avoid printinf warning. Thanks to Sebastian Fischer for the help. - Allow null value in BFILE to the oar2pg_get_bfilename(). - Update documentation about BFILE export. - Add drop function ora2pg_get_bfilename() when necessary. - Add support to BFILE external path export by creating a function ora2pg_get_bfilename( p_bfile IN BFILE ) to retrieve path from BFILE. BFILE will be exported as text field with the full path to the file as value. Note that this is the first time that Ora2Pg need write access to the Oracle database, if you do not have BFILE or you have set the corresponding PostgreSQL type asd bytea (the default) the function will not be created. - Fix a performance issue when extracting BLOB with a LongReadLen upper than 1MB. - Fix priviledge on schema created from Oracle package body. Thanks to Dominique Legendre for the report. - Add object type in comment before priviledge extraction. - Order output of grant to groups grants by object types. This is useful to quickly disable some SQL orders corresponding of not already loaded objects. Thanks to Dominique Legendre for the feature request. - Fix progress bar output. - Fix priviledge on sequence, tablespace and schema. - Fix backward compatibility with Oracle 8i, remove query with JOIN. Thanks to Sebastian Fischer for the report. - Fix backward compatibility with Oracle 8i on priviledge extraction. Thanks to Sebastian Fischer for the report. - Fix backward compatibility with Oracle 8i on index extraction. Thanks to Sebastian Fischer for the report. - Add more precision in cost estimation. - Add somme other PL/SQL uncovered code detection. - Add more debug information during data extraction. - Removed progress bar when debug is enabled. - Add report and estimate cost about CHECK constraint and function based indexes. - Update documentation about new export directives SHOW_REPORT and ESTIMATE_COST. - Add --estimate_cost and --cost_unit_value command line options. - Add ESTIMATE_COST and COST_UNIT_VALUE to default configuration file. - Rewritte and extend support to ROWNUM replacement. - Remove incompatible grants between Oracle and the PortgreSQL export, especially on views. - Limit GRANT export to VALID object. Activate EXPORT_INVALID to enable grants export on all object. - Add export of VALID only views. To export all with INVALID ones you must activate the EXPORT_INVALID directive. Thanks to Dominique Legendre for the feature request. - Fix issue in substr() pl/sql replacement, thanks to Dominique Legendre for the report, plus add other code replacements in pl/sql. - Fix issue with function name not on the same line as the create statement - was affecting file input only. - Add report of number of JOB object in the database (SHOW_REPORT). - Add PL/SQL replacement of various form of EXEC function call. - Remove creation of password with users that are not requiring password. Thanks to Dominique Legendre for the feature request. - A sql type and a precision can now be used in REPLACE_AS_BOOLEAN to replace all filed with that type as a boolean, example: NUMBER:1 will replace all field of type NUMBER(1) as a boolean. - Fix grants on partition export, will now used all_ and user_ tables. - Fix removing of newline in the DECLARE clause. Thanks to Dominique Legendre for the report. - PostgreSQL client_encoding is now forced to UTF8 when BINMODE is set to utf8. Thanks to Dominique Legendre for the report. - Replace DISABLE TRIGGER ALL by DISABLE TRIGGER USER following the value if USER_GRANTS to avoid permission denied on constraint trigger when data are load under a non PG superuser. Thanks to Dominique Legendre for the report. - Rename DISABLE_TABLE_TRIGGERS to DISABLE_TRIGGERS and set default value to 0. Other values are USER or ALL following the connected user. - Fix missing newline after comment in PL/SQL code. Thanks to Dominique Legendre for the report. - Fix report message on external table export. - The export TYPE have been entirely rewritten to only export supported user defined types. Exported are: Nested Tables, Object type, Type in herited and Subtype, Varrays. Associative Arrays, Type Body and type with member method are not supported. - When FILE_PER_INDEX is enable, SQL order to move indexes in their respective tablespace will be written into a dedicated file prefixed by TBSP_INDEXES_. - Fix location on external table export. Thanks to Thomas Reiss for the help. - PG_SUPPORTS_INSTEADOF is now activated by default, that mean that migration should be done on PG >= 9.1. - Remove obsolete --xtable commande line option, should be replaced by --allow, backward compatibility is preserved. - Add EXTERNAL_TO_FDW configuration directive, disable by default, to export all Oracle external tables as file_fdw foreign tables. - Fix an other case where user defined type were not exported with an ending semi-colon. Thank to Dominique Legrendre for the report. - Fix export of user defined type with extra ");" at end of the type definition and remove system types from export. Thanks to Dominique Legendre for the report. - Add PLSQL replacemement of currval. Thanks to Thomas Reiss for the patch. - Add PLSQL replacement of PIPELINED and PIPE ROW by SETOF and RETURN NEXT. - Add rewrite of Oracle DETERMINISTIC function into PostgreSQL IMMUTABLE function. - Fix copy during install on MacOSx and add /Y option to windows install copy to force overwrite existing files. Thanks to Dennis Spaag for the report. - Fix issue exporting rows with perl ARRAYS ref. Thanks to Sorin Gheorghiu for the report. - Add report of number of database link in SHOW_REPORT output. - Fix major bug on export of NUMBER with precision, they was all exported as bigint. Thanks to Dominique Legendre for the report. - Add progress bar during SHOW_REPORT export. - Add detailed report about index in SHOW_REPORT output. - Fix data export when schema was given in lower case. Thanks to Dominique Legendre for the report. - Add SHOW_REPORT export type to display a full summary of the Oracle database content. - PLPGSQL: add the name keyword to XMLELEMENT call. Thanks to Thomas Reiss for the hint. - Add SHOW_VERSION export type to display the version of Oracle. - Add COLLATION to the keyword list. Thanks to Dominique Legendre for the report - Change documentation to add more detail on exporting Oracle views as PostgreSQL tables based on the new VIEW_AS_TABLE directive. - Add -a | --allow option and --view_as_table to ora2pg script. - Add VIEW_AS_TABLE configuration option to allow export of view as table and permit the additional use of the ALLOW or/and EXCLUDE directive. Thanks to Dominique Legendre for the feature request. - Removed conflict with transaction when DEFER_FKEY was enabled and allow DEFER_FKEY and DROP_FKEY to be enabled both. Before, only DEFER_FKEY was used in this case, now both are used and of course DEFER_FKEY is wasted. Thanks to Dominique Legendre for the report. - Directives ALLOW and EXCLUDE are now usable with all kind of object following the export type. - Rename TABLES directive as ALLOW to be less confusing, backward compatibility is preserved. - Thanks to Dominique Legendre for the feature request. - Remove auto ordering of table export following the foreign keys to fix an infinite loop. Thanks to Siva Janamanchi for the report. - Rewrite the view as table export to reuse the same code as table export, old code was resulting in issues with disable triggers and deferring constraints. - Remove alter session to set NLS_NCHAR that was returning error on some installation. Thanks to Dominique Legendre for the report. - Fix replacement of IS SELECT wrongly replaced by IS PERFORM in some case. Thanks to Dominique Legendre fot the report. UPGRADE: Almost all files have changed so a new installation is required. 2012 10 07 - v9.3 - Add auto detection of Oracle character set and the corresponding PostgreSQL client encoding to use. NLS_LANG and CLIENT_ENCODING configuration directives can be leaved commented, Ora2Pg will set their values automatically. - Add PL/SQL replacement of CURSOR IS SELECT by CURSOR FOR SELECT and IS REF CURSOR by REFCURSOR. Thanks to Dominique Legendre for the report. - Fix missing set client_encoding orders into fonction or procedure export file. Thanks to Dominique Legendre for the report. - Fix not working SKIP configuration directive. Thanks to Siva Janamanchi for the report. - Add configuration directive NULL_EQUAL_EMPTY to disable the IS NULL and IS NOT NULL replacement into PL/SQL code. Enabled by default. Thanks to Dominique Legendre for the feature request. - Remove exclusion of object names with the dollar sign. Thanks to Konrad Beiske for the suggestion. - Fix timestamp with time zone when microsecond is enabled. Thanks to Siva Janamanchi for the report. - Fix extra semi-column when PKEY_IN_CREATE is enabled. Thanks to Siva Janamanchi for the report. - Update configuration about boolean replacement. - Allow any column type replacement as a boolean in PostgreSQL, values will be converted as well. Thanks to Konrad Beiske for the feature request. - Add REPLACE_AS_BOOLEAN and BOOLEAN_VALUES configuration directives to allow type replacement with a boolean. Thanks to Konrad Beiske for the feature request. - Add new configuration directive PKEY_IN_CREATE to add primary keys definition in CREATE TABLE statement instead of creating them after with an ALTER TABLE statement. For Greenplum database, primary key must be created with the CREATE TABLE statement so you may want to enable this configuration directive. Thanks to Siva Janamanchi for the feature request. - Add new configuration directive USE_RESERVED_WORDS to force Ora2Pg to auto-detect PostgreSQL reserved words in Oracle object's names and automatically double quote them. Thanks to Siva Janamanchi for the feature request. - SHOW_TABLE and SHOW_COLUMN will now display a warning when Oracle object's name is a PG reserved words. Those names will need to be renamed or double-quoted (see USE_RESERVED_WORDS). - Add TIMESTAMP WITH LOCAL TIME ZONE Oracle type conversion to timestamp and force timestamp with time zone format to use TZH:TZM. Thanks to Siva Janamanchi for the report. - Fix table and column replacement issues introduced with path that removed double-quote when PRESERVE_CASE is disabled. Thanks to Steve DeLong for the report. - PLPGSQL convertion: Fix SELECT replacement with PERFORM in VIEW declaration. Thanks to Thierry Capitaine for the report. - Add display Ora2Pg type conversion map between Oracle originals types and PostgreSQL's types when using export type SHOW_COLUMN. Thanks to Thierry Capitaine for the feature request. - Reorder command line options in ora2pg script usage and documentation. - Add call to quote_ident() and quote_literal() into materialized functions to secure parameters. - Fix major issue in pl/sql to pl/pgsql conversion with multiple package declaration in the same code record. Thanks to Marc Cousin for the report. - Add data type TIMESTAMP WITH TIME ZONE. Thanks to Siva Janamanchi for the report. - Add new export type: MVIEW to allow export of all materialized views as snapshot materialized view (fully reload of the view). - Add -e | --exclude option to Perl script ora2pg to exclude some given objects from the export. It will override any value of the EXCLUDE directive. The value is a coma separated list of object name or regex. - Update domumentation about the EXCLUDE directive change. - Allow exclusion from export of functions, procedures and functions in package by specifying a list of name or regex in EXCLUDE directive. Thanks to Keith Fiske from Omniti for the feature request. UPGRADE: Almost all files have changed so a new installation is required. 2012 09 05 - v9.2 - In plpgsql conversion, SELECT without INTO becomes PERFORM. - In plpgsql conversion, EXECUTE IMMEDIATE replaced by EXECUTE. - Fix DATA_TYPE value in configuration file. - Fix case sensitivity on data export using COPY mode. - Directive XML_PRETTY is now disabled by default as it is better to use getClobVal() to get the XML data out of an xmltype column. - Add documentation about regex usage int EXCLUDE and TABLES directives. - Remove all double-quote around object name when CASE_SENSITIVY is disabled. Thanks to Dominique Legendre for the suggestion. - Rename CASE_SENSITIVE by PRESERVE_CASE to be less confusing, backward compatibility preserved. Thanks to Dominique Legendre for the request. - Add support to user defined type data export. Before it will simply export an array reference ARRAY(0xa555fb8), now the array is explored and inserted as ROW(col1,col2,...). Thanks to Mathieu Wingel for the feature request. - Fix bug in direct data import in postgresql with COPY: pg_putcopydata can only be called directly after issuing a COPY FROM command. Thanks to Steve Delong for the report. - Add warning at any debug level before abort when a data file already exist during data export. - Fix issue with oracle sensitivity when exporting data. - Fix search_path on package export, indexed and constraints files on TABLE export. - Remove obsolete ORA_SENSITIVE configuration directive, thanks to Dominique Legendre it is no more used. - Force automatic conversion of PL/SQL when entry is an input file. - Fix errors in main file for package loader with FILE_PER_FUNCTION enabled. - Fix case where package body where not exported. - Add missing EXPORT_INVALID directive into default configuration file. - Fix replacement of END followed by the name of the function, the semi- colon was removed. - Fix case sensitivity issue in INDEX creation. - Fix case sensitivity issue in CHECK constraint and a typo in a progress bar variable. - Replace old DATA export type by INSERT in configuration file. - Fix case sensitivity issue in ALTER TABLE ... ADD CONSTRAINT. Thanks to David Greco for the report. - Add set client_encoding before table output to handle character encoding into comments and possibly objects names. - Fix some case sensitivity issue with schema name. Thanks to Dominique Legendre for the report. - Do not display warning message about direct import if no connection to a PostgreSQL database is defined. - Allow multiple export type to be specified into the ora2pg -t command line option. - Dump progress bar to stderr instead of stdout to separate logs. - Add new -q | --quiet option to perl script ora2pg to disable progress bar. 2012 08 19 - 9.1 - Add progress bar to show data export progression. - Add -q | --quiet option to ora2pg perl script to disable progress bar. - Change documention about tnsnames.ora to mark it is not necessary. - Add progress bar during data export, per table and globaly. - Replace export type DATA by INSERT to mark the difference with COPY and avoid confusion. Documentation is updated and full backward compatibility preserved. - Improve Oracle case sensitivity detection on column and update documentation about ORA_SENSITIVE directive - Direct import for COPY statement now used DBD::Pg and pg_putcopydata() instead of a pipe to psql command. - Fix case sentitivity issue on disabling/enabling all triggers. - Add autodetection of case sensitivity with column name. - Move trunc() to data_truc() convertion into the ALLOW_CODE_BREAK part. - Update comment about FILE_PER_FUNCTION in configuration file. - Fix NOT NULL constraint add twice, the first time in the column definition and the second time in an ALTER TABLE ... ADD CONSTRAINT ... CHECK ( ... NOT NULL). Reported by Dominique Legendre. - Add support to direct CALL of stored procedures in trigger definition. Reported by Dominique Legendre. - Remove index creation on primary and unique key autogenerated by PostgreSQL. - Fix PL/SQL to PLPGSQL automatic convertion on index when exporting data with DROP_INDEX activated. - Fix DROP_INDEX to only delete indexes that will be created at end. - Fix search path when exporting data with EXPORT_SCHEMA disabled - Add missing documentation about the LOGFILE directive - Fix case sensitivity on sequence export. They will now always be insensitive as in PostgreSQL its called is converted between quotes: nextval('seq_name'). Reported by Dominique Legendre. - Limit export of primary and unique key if KEEP_PKEY_NAMES is enabled to those who are not autogenerated by Oracle. Reported by Dominique Legendre. - Trigger export is now limited to those belonging to table that are not excluded from the export (see TABLES and EXCLUDE directives). Reported by Dominique Legendre - Fix case sensitivity on trigger export. - Fix data export failure in table with column name with accent. Reported by Dominique Legendre. - Fix set client_encoding syntax. Reported by Dominique Legendre - Add automatic try with oracle sensitivity when an error occurs during retreving table information. This additionaly also fixes an error when table has accent on his name. - Fix replacement of user defined data type with directive DATA_TYPE. Reported by Dominique Legendre. - Fix function or procedure detection with external input file. Reported by Arul Shaji. - Update documentation about Windows installation and ActiveState Perl distribution. Thanks to Stephan Hilb for the report. - Fix date format issue by forcing NLS_DATE_FORMAT to format: YYYY-MM-DD HH24:MI:SS. Thanks to Stephan Hilb for the report. - Remove obsolete pod documentation in Ora2Pg.pm. - Add new configuration directive CREATE_SCHEMA to disable the sql command of schema creation at top of the output file during TABLE export type. Patch by David Greco. - Added converting INSERTING/UPDATING/DELETING to PG_OP=INSERT, etc. Patch by David Greco. - Fix parsing leading ':' on triggers, as they generally have :NEW and :OLD to denote the new and old recordsets. Patch by David Greco - Add new PG_INTEGER_TYPE configuration directive activated by default, to limit conversion into postgresql integer or bigint of Oracle number without scale - NUMBER(p), PG_NUMERIC_TYPE is now reserved to convert NUMBER(p,s). Patch by David Greco. - Limit numeric with precision <= 9 to be converted as integer, numeric with precision >= 10 will be converted to bigint to handle integer above 2147483647. Patch by David Greco. - Add plsql to plpgsql automatic conversion on check constraints. Patch by David Greco. - Add plpgsql replacement, patch by David Greco: REGEX_LIKE( string, pattern ) => string ~ pattern - Update documentation about NOESCAPE and STANDARD_CONFORMING_STRING - Change place of the ENABLE_MICROSECOND into the documentation. - Fix forgot to add documentation about encryption with Oracle server. - Add missing DISABLE_COMMENT configuraton directive in default configuration file and update documentation 2012 07 15 - 9.0 - Remove call to obsolete BINDIR and MANDIR variables into packaging scripts to reflect the changes in Makefile.PL. - Update documentation about installation of Ora2Pg under Windows. - Automatically set LONGREADLEN to ORA_PIECE_SIZE length if the last one is larger, for CLOB export. - Change Makefile.PL and source tree to fully support installation under Windows OSes. - Change double quote by single in Makefile.PL perl replacement call. - Replace double quote by single one in $CONFIG_FILE default setting to simplify automatic replacement at install. - Fix CLOB export that was limited to 64Kb even with LONGREADLEN defined to an upper value. Patch use the ora_piece_size DBD::Oracle prepare attribute. Patch by Mohamed Gargouri. See here for more detail: http://search.cpan.org/~pythian/DBD-Oracle-1.46/lib/DBD/Oracle.pm#Piecewise_Fetch_with_Polling - Add a note into documentation about encrypted communication between Ora2Pg and Oracle. Note by Jenny Palomino. - Change documentation to reflect change to the format of the Oracle timestamp with millisecond. This format is now enabled by default in configuration file. - Fix bug with LONGREADLEN and LONGTRUNCOK when exporting LOB that was not applied even after change into the configuration file. Reported by Mohamed Gargouri - Fix microsecond format FF3 not compatible with Oracle 8i. Set to FF. - Add a warning to stderr when a table export need that ORA_SENSITIVE be enabled. - Fix case where Oracle indexes with same name as a constraint was not exported - Rodrigo The following are old patches that was not applied to v8.10 and the git repository: - Fix creation of bad constraint for each indexes. - Add DISABLE_COMMENT configuration directive to remove comments from TABLE export type. Comments are exported by default. - Fix a bug in removing function name repetion at end - Add PL/SQL to PLGPSQL replacement of the to_number function - Fix PL/SQL to PLGPSQL replacement of substr into substring - Add replacement of specials IEEE 754 values BINARY_(FLOAT|DOUBLE)_NAN and BINARY_(FLOAT|DOUBLE)_INFINITY by NaN and Infinity on PLPGSQL conversion and on data export - Thanks to Daniel Lyons. - Fix return type of function with just OUT or INOUT params. Thanks to Krasi Zlatev for the patch. - Add schema name on functions or procedures export when EXPORT_SCHEMA is activated. Thanks to Krasi Zlatev for the patch. - Fix case sensitivity issue with schema on partition export. - Fix case sensitivity issue with --xtable option. - Fix issues with case sensitivity on the schema owner set into the SCHEMA configuration directive. - Add default search_path on schema for contraints, index and data export when EXPORT_SCHEMA is activated. - Fix case sensitivity issue in search_path. - Force Oracle datetime format to be YYYY-MM-DD HH24:MI:SS.FF in client session to prevent other defined output format. Thanks to Aaron Culich for the patch. - Add export/import of table and column comment. Thanks to Krasi Zlatev for the patch. 2012 06 26 - 8.13 - Fix broken export with missing single quote in Oracle timestamp export formating with to_char(timestampcolumn, YYYY-MM-DD HH24:mi:ss). Thanks to Steve Delong for the report. 2012 06 22 - 8.12 - Add nex configuration directive ENABLE_MICROSECOND to allow timestamp to be exported with millisecond precision. Thanks to Patrick King for the feature request. - Fix multiple quote on foreign keys column names. Thanks to Vitaliy for the report. - Add new export type FDW to allow table export as foreign table for oracle_fdw. Thanks to David Fetter for the feature request. - Fix typo in LongTruncOk variable name. Thanks to Magnus Hagander for the patch. - Add XML_PRETTY configuration directive to replace getStringVal() by getClobVal() when extracting XML data. Thanks to Magnus Hagander for the patch. - Fix case sensitivity issue in ALTER TABLE and TRUNCATE statement. Thanks to Magnus Hagander for the patch. UPGRADE: Ora2Pg.pm and ora2pg perl scripts have changed as well as configuration file. Documentation has been updated too so you'd better install all again. 2012 04 30 - 8.11 - Fix an error when running ora2pg directly against PG, index and constraints are created against PG instead of being written to the output file. Thanks to David Greco for the report. - Ora2Pg will now output a warning message when direct import to PG is set with other import type than COPY and DATA. - Fix NUL character removing on LOB to bytea export. Thanks to info31 on PostgresqlFr for the report. 2012 03 11 - 8.10 - Add two configuration directives to control the BLOB export. LONGREADLEN to set the database handle's 'LongReadLen' attribute to a value that will be the larger than the expected size of the LOB. LONGTRUNKOK to bypass the 'ORA-24345: A Truncation' error. Thanks to Dirk Treger for the report. - Fix install problem on non-threaded Perl and the threads package. Replace use() by require() call. Thanks to Ian Sillitoe for the patch. - Fix strange Oracle behaviour where binary_double infinity is exported from Oracle as '~'. Replaced by 'inf'. Thanks to Daniel Lyons for the report. UPGRADE: only Ora2Pg.pm have changed so you can just override it. See also documentation for new configuration directives: LONGREADLEN and LONGTRUNKOK. 2011 11 07 - 8.9 - Fix double quote into file name of package function export when case sensitivity is preserved. - Add support to XMLType data extraction. Thanks to Aaron Culich for the report. Before this release, xml data was exported as a Perl array reference. - Fix bug during foreign key export when foreign keys have different owners. Thanks to Krasi Zlatev for the patch. - Add support to plpgsql conversion during index extraction as many index use some Oracle function on their declaration. Thanks to Sriram Chandrasekaran fot the feature request. - PLSQL: Add replacement of Oracle subtr() by PostgreSQL substring(). Thanks to Sriram Chandrasekaran fot the feature request. - PLSQL: Add replacement of Oracle decode() by PostgreSQL CASE/THEN/ELSE. Thanks to Sriram Chandrasekaran fot the feature request. Note that this two replacement was not implemented because they could break the code if there's complex subqueries inside their declaration. This is why you can enable it by setting ALLOW_CODE_BREAK to 1 (new). In later release this directive will be enable by default. - Add output ordering on some object name so that results between two runs can be compared. Thanks to Henk Enting for the patch. - Fix misshandling of all cases of RAISE_APPLICATION_ERROR rewrite into RAISE EXCEPTION concatenations. Thanks to Krasi Zlatev for the report. UPGRADE: only Ora2Pg.pm and Ora2Pg/PSQL.pm have changed so you can just override them if you dont want to reinstall all. 2011 10 13 - 8.8 - Before that release when you upgraded Ora2Pg using Makefile, the old ora2pg.conf was renamed as ora2pg.old. This can lead to lost previous configuration, the old ora2pg.conf is now untouched and the new one is installed as ora2pg.conf.new. - Renamed ora2pg.pl into ora2pg_pl in the package before installation to avoid the copy of the perl script into the site Perl install dir. It is still installed as ora2pg in /usr/local/bin by default. - Fix errors that appeared to be due to no quoting on the field names when ORA_SENSITIVE is enabled. Thank to Sam Nelson for the patch. - Limit case sensitivity on check constraints to column names only, before that if there was a value between double quote into the check constraint, it was wrongly changed to lower case. - Fix broken case sensitivity at data export when disabling/enabling triggers and truncating tables with copy or insert statement. - Change Ora2Pg version in packaging files that was still in 8.5. UPGRADE: only Ora2Pg.pm have changed so you can just override it. 2011 09 07 - 8.7 - The escape_bytea() function has been rewritten using a prebuild array to gain twice of performances. Thanks to Marc Cousin from Dalibo for the patch. - Improve speed of bulkload data by disabling autocommit by issuing a BEGIN at the start and COMMIT at the end. - Add multi-threading support. It is only used to do the escaping to convert LOBs to byteas, as it is very cpu hungry. There's a lot of CPU-waste here. The threads number is controlled by a new configuration directive: THREAD_COUNT. Putting 6 threads will only triple your throughput, if your machine has enough cores. If zero (default value), do not use threads, do not waste CPU, but be slower with bytea. Performance seems to peak at 5 threads, if you have enough cores, and triples throughput on tables having LOB. Another important thing: because of the way threading works in perl, threads consume a lot of memory. Put a low (5000 for instance) DATA_LIMIT if you activate threading. Many thanks to Marc Cousin for this great patch. - Fix standard_conforming_string usage on export as INSERT statement. - Fix an issue with importing Oracle NULL character (\0 or char(0)) with bytea and character data with UTF8 encoding. Now whatever is the data type or the encoding, this character is simply removed to prevent the well known 'ERROR: invalid byte sequence for encoding "UTF8": 0x00.' Thanks to Jean-Paul Argudo from Dalibo for the report. - Fix an incorrect syntax for "for each statement" triggers. Thanks to Henk Enting for the report. - Add comment at end of line to signal on which cursor the replacement on " EXIT WHEN (...)%NOTFOUND " is done. This will return something like "IF NOT FOUND THEN EXIT; END IF; -- apply on $1". Thanks to jehan Guillaume De Rorthais from Dalibo for the report this help a lot during Pl/Pgsql code review. - Fix table/column name replacement on constraint creation and dropping when REPLACE_TABLES/REPLACE_COLS is set during DATA/COPY export. - Fix table/column name replacement on indexes creation and dropping when REPLACE_TABLES/REPLACE_COLS is set during DATA/COPY export. - Remove unused table name parameter in _drop_indexes() function. - Add support to REPLACE_TABLES/REPLACE_COLS during schema export. Before this release those replacements were only applied to DATA or COPY export. You can now use it in schema export, it will replace table and/or column names in the TABLE/INDEX/CONSTRAINT schema export. MODIFY_STRUCT is still limited to DATA or COPY export as it have no sense outside this export. Unfortunately those replacements can not be done easilly in other export type like TRIGGER, FUNCTION, etc. so you must still edit this code by hand. - Use the bundled Perl Config module to detect if Perl is compiled with useithread. This mean that the old local defined %Config hash has been replaced by %AConfig. - SKIP indices is now obsolete and must be replaced with SKIP indexes. backward compatibility is preserved. - The file generated when FILE_PER_INDEX is activated has been renamed into INDEXES_... instead of INDICES_... - Add a warning on tablespace export when Oracle user is not a dba. - Fix fatal error when dumping to one file per function with double directory output. - Fix double print of FATAL messages and dirty database disconnect on fatal errors. - Add setting of client_encoding into each export type as defined in the configuration file. - Update web site documentation. UPGRADE: Ora2Pg.pm, Ora2Pg/PGSQL.pm and ora2pg have changed so they must be overwritten. There's also changes in the configuration file and documentation has changed as well. Backward compatibility is fully preserved. 2011 07 07 - 8.6 - Remove "use strict" from head of Ora2Pg.pm that breaks view export. This is usually removed before public release, but not this time. Thanks to Jehan Guillaume de Rorthais from Dalibo for the report. - Add a new configuration directive called COMPILE_SCHEMA that force Oracle to compile the PL/SQL before code extraction to validate code that was invalidate for any reason before. If you set it to 1, you will force compiling of the user session schema, but you can specify the name of the schema to compile as the value too. Thanks to Jean-Paul Argudo from Dalibo for the solution. - Add new configuration directive EXPORT_INVALID to allow export of all PL/SQL code even if it is marked as invalid status. The 'VALID' or 'INVALID' status applies to functions, procedures, packages and user defined types. - Excluded from export all tables, types, views, functions and packages that contains a $ character. Most of the time they don't need to be exported. - PLSQL: add automatic conversion of Oracle SYS_REFURSOR as PostgreSQL REFCURSOR. - Rewrite entirely the parser of DBMS_OUTPUT du to concatenation errors. - PLSQL: add automatic replacement of some Oracle exception errors: INVALID_CURSOR=>INVALID_CURSOR_STATE, ZERO_DIVIDE=>DIVISION_BY_ZERO, STORAGE_ERROR=>OUT_OF_MEMORY. UPGRADE: Ora2Pg.pm and Ora2Pg/PGSQL.pm have changed so they must be overwritten. There's also changes in the configuration file and documentation has changed as well. Backward compatibility is fully preserved. 2011 07 01 - 8.5 - When FILE_PER_FUNCTION is activated and export type is PACKAGE, Ora2Pg will now save all functions/procedures of a package body into a directory named as the package name and into different files. This will allow to load each function separatly or load them all with the OUTPUT SQL script generated by Ora2Pg. - Fix Oracle package body parsing failure when a procedure is declared inside an other. - Add new configuration options FILE_PER_CONSTRAINT and FILE_PER_INDEX to generate three files during the schema extraction. One for the 'CREATE TABLE' statements, one for the constraints (primary keys, foreign keys, etc.) and the last one for indices. Thanks to Daniel Scott for the feature request. - Allow to process PL/SQL Oracle code from file instead of a database to apply Ora2Pg code conversion. Thank to Mindy Markowitz for the feature request. See -i or --input_file command line option to ora2pg perl script or INPUT_FILE new configuration option. - Add new configuration directive STANDARD_CONFORMING_STRINGS that is used only during DATA export type to build INSERT statements. This should avoid 'WARNING: nonstandard use of \\ in a string literal'. Please check that this behavior is backward compatible with your PostgreSQL usage as this is enabled by default now. UPGRADE: The new features has changed Ora2Pg.pm and ora2pg.pl so that they must be overwritten. There's also changes in the configuration file and documentation has changed as well. Take care of backward compatibility with escaped strings in DATA export type and the new behavior on PACKAGE export. 2011 06 07 - 8.4 - Moves Ora2Pg to SourceForge.net. - Fix an issue on setting owner in "ALTER SEQUENCE ... SET OWNER TO". Thanks to Herve Girres for the report. - Bugfix on lower case convertion for check constraints extraction. Thanks to Alexander Korotkov for the patch. UPGRADE: There's no new functionality, this is a bug fix release. 2011 05 11 - 8.3 - Fix issue on inherited user defined types converted to inherited tables. Add comment on unsupported inherited type in PostgreSQL too. Thanks to Mathieu Wingel for the report. - Fix issue on column default values. Oracle all this kind of strange syntax: counter NUMBER(4) default '' not null, that was translated to counter smallint DEFAULT '' by Ora2Pg. Thanks to Mathieu Wingel this is now rewritten as DEFAULT NOT NULL. - Fix case sensitivity on create view when there was double quote on the column name statement part. Thanks to Mathieu Wingel or the report. - Fix bad patch applied on column name case sensitivity issue during check constraint export. Thanks to Philippe Rimbault for the report. - Fix bug on package export introduced into version v8.2. The issue was related to end of package procedure detection. hanks to Mathieu Wingel or the report. UPGRADE: There's no new functionality, this is a bug fix release and every one should upgrade to it. 2011 05 01 - 8.2 - PLSQL: automatic replacement of EXIT WHEN cursor%NOTFOUND; by Pg synthax: IF NOT FOUND THEN EXIT; END IF;. Works with additional condition too. - PLSQL: Automatic replacement of SQL%NOTFOUND by NOT FOUND. - PLSQL: Add detection of TOO_MANY_ROW to NO_DATA_FOUND to add STRICT. - Completely rewrite the parsing of Oracle package body to handle all cases and especially prodedure declared into an other procedure. Those procedure are renamed INTERNAL_FUNCTION and must be rewritten. - Fix type usage of ora2pg Perl script. - Add a new directive FORCE_OWNER. By default the owner of the database objects is the one you're using to connect to PostgreSQL. If you use an other user (postgres for exemple) you can force Ora2Pg to set the object owner to be the one used in the Oracle database by setting the directive to 1, or to a completely different username by setting the directive value to that username. Thanks to Herve Girres from Meteo France for the suggestion and patch. - Add --forceowner or -f command line option to ora2pg program. - Add SHOW_ENCODING extract type to return the Oracle session encoding. For example it can return: NLS_LANG AMERICAN_AMERICA.AL32UTF8 - Remove SYS_EXTRACT_UTC from index creation as Pg always stores them in UTC. Thanks to Daniel Scott for the patch. - In PLSQL code SYS_EXTRACT_UTC is replaced by the Pg syntaxe: field AT TIME ZONE 'UTC'. - Fix a pending problem with "Wide character in print at" on COPY mode. Thanks to Bernd Helmle from Credativ GmbH for the patch. - PLSQL: Add automatic rewrite of FOR ... IN REVERSE ... into Pg synthax - Fix column name case sensitivity issue during check constraint export. Thanks to Daniel Berger for the report. - Remove the possibility to add comment after a configuration directive it may not be used and it was generating an issue with the passwords configuration directives for examples. Thanks to Daniel Berger for the report. - Complete rewrite of user defined type extraction. Add support of inherited type using Oracle UNDER keyword as well as better support to custom type with BODY. Thanks to Mathieu Wingel for the report. - Fix case sensitivity on user defined types. Thanks to Mathieu Wingel for the report. UPGRADE: All files have changed so you need a fresh install/upgrade. Previous release used to remove any string starting from a # in the config file, this was to allow comments after a configuration directive. This possibility have been removed in this release so you can no more add comments after a configuration directive. 2011 03 28 - 8.1 - Prevent Ora2PG to export twice datas when using FILE_PER_TABLE and the data output file exist. This is useful in case of export failure and you don't want to export all data again. This also mean that if you want to get new data you have to remove the old files before. - Fix parsing of procedure/function into pl/sql Oracle package. - Fix bug in IS NULL/IS NOT NULL replacement. Thanks to Jean-Paul Argudo from Dalibo for the report. - Add CREATE OR REPLACE on RULE creation. - Add DROP TRIGGER IF EXISTS before trigger creation. - Replace Oracle date "0000-00-00" by NULL. - Fix infinite loop in package/fonction type replacement. - Add one file per package creation if FILE_PER_FUNCTION is enabled. - Fix double quote in name of custom type extraction. - Add extraction of custom type IS VARRAY as an custom type of table array. Thank to Jean-Paul Argudo from Dalibo for the patch. - Fix multiple double quote in name of create index definition. - Apply excluded and limited table to tablespace extraction. - Fix function and procedure detection/parsing on package content. - Fix schema prefix in function name declaration in package export. - PLSQL: Replace some way of extracting date part of a date : TO_NUMBER(TO_CHAR(...)) rewritten into TO_CHAR(...)::integer when TO_NUMBER just have one argument. - Fix Makefile.pl error when trying to modify file ora2pg now renamed into ora2pg.pl - Add 3 new export types SHOW_SCHEMA, SHOW_TABLE and SHOW_COLUMN. Those new extraction keyword are use to only display the requested information and exit. This allow you to quickly know on what you are going to work. The SHOW_COLUMN allow a new ora2pg command line option: '--xtable relname' or '-x relname' to limit the displayed information to the given table. - Add type replacement for BINARY_INTEGER and PLS_INTEGER as integer. UPGRADE: Please make a full upgrade asap to this release. 2011 03 15 - 8.0 This major release simplify and improve Oracle to PostgreSQL export. Ora2Pg v8.x now assume that you have a modern PostgreSQL release to take full advantage of the Oracle compatibility effort of the PostgreSQL development team. Ora2Pg since v8.x release will only be compatible with Pg >= 8.4. - Remove addition of AS for alias as with modern PG version this can be optional (Pg >= 8.4). - Fix CREATE with missing USER/ROLE for grant extraction. Thanks to Herve Girres for the report. - Apply missing psql_pgsql converter to view definition. - PLSQL : Normalize HAVING ... GROUP BY into GROUP BY ... HAVING clause - PLSQL : Convert call to Oracle function add_months() in Pg syntax - PLSQL : Convert call to Oracle function add_years() in Pg syntax - Apply missing psql_pgsql converter to triggers WHEN clause. - Fix DECLARE CURSOR rewrite. - Allow one file per function / procedure / package exported with a new configuration option FILE_PER_FUNCTION. Useful to editing and testing. Thank to Jean-Paul Argudo from DALIBO for the feature request. - The FILE_PER_TABLE configuration option is now also applied to views. - Remove obsolete PG_SUPPORTS_INOUT as it is supported by with modern PG version (Pg >= 8.4). - Remove obsolete PG_SUPPORTS_DEFAULT as it is supported by with modern PG version (Pg >= 8.4). - Allow to adjust PostgreSQL client encoding with a new configuration directive: CLIENT_ENCODING. - Add TRUNCATE_TABLE configuration directive to add TRUNCATE TABLE instruction before loading data. - Add type conversion of Oracle XMLTYPE into PostgreSQL xml type. - PLSQL: SYSDATE is now replaced by LOCALTIMESTAMP to not use timezone. Thanks to Jean-Paul Argudo from DALIBO for the report. - Use 'CREATE OR REPLACE' on create trigger function instruction. - Fix prefixing by OUTPUT_DIR when file per table/function is enabled. - Use 'CREATE OR REPLACE' on create view. - PLSQL_PGSQL is now enabled by default. If you want to export Oracle original function/procedure/package code, disable it. - PLSQL: WHERE|AND ROWNUM = N; is automatically replaced by LIMIT N; - PLSQL: Rewrite comment in CASE between WHEN and THEN that makes Pg parser unhappy. - PLSQL: Replace SQLCODE by SQLSTATE UPGRADE: You must reinstall all and review your configuration file 2011 02 14 - 7.3 - Remove PG_SUPPORTS_INOUT, now Ora2Pg assumes the PostgreSQL database destination support it (Pg > v8.1). - Remove PG_SUPPORT_ROLES, now Ora2Pg assumes the PostgreSQL database destination support it (Pg > v8.1). - Complete rewrite of the GRANT (user/role/grant) export type. It now should be only related to the current Oracle database. Note that do not try to import rights asis as you may import have errors or worse miss handling of the rights! Just remember that for example in Oracle a schema is nothing else than a user so it must not be imported like this. - Fix multiple errors in partitionning definition. Thank to Reto Buchli for the report. - PLSQL: reordering cursor Oracle declaration "DECLARE CURSOR name" into "DECLARE name CURSOR". Thank to Reto Buchli (WSL IT) for the report. - Fix miss handling of DEFAULT parameters value in convert_function(). Thanks to Leonardo Cezar for the patch. - Fix Oracle tablespace export where Pg tablespace location was based on Oracle filename. This fix extract the path and replace the filename with tablespace name. Thank to Reto Buchli (WSL IT) for the report. - Fix parsing of ending function code. Thanks to Leonardo Cezar for the patch. - Fix call to _convert_procedure() that is in fact the same function as _convert_function(). Thanks to Leonardo Cezar for the report. - Fix multiple call on tablespace alter index on the same object. Thank to Reto Buchli (WSL IT) for the report. - PSQL: Rewrite RAISE EXCEPTION concatenations. Double pipe (||) are replaced by % and value is set as parameter a la sprintf. Thank to Reto Buchli (WSL IT) for the report. - Add missing comment of PARTITION export type into configutation file. - Complete rewrite of the table partition export part has it was not handling all case and was really buggy. - PLSQL: add normalisation of the to_date() function. - Ora2Pg now warns during grant export when it is not connected as an Oracle DBA user. GRANT export need rights of Oracle DBA or it fail. - Fix install of changelog into Makefile.PL, name was wrong. Thanks to Julian Moreno Patino for the patch. 2011 01 12 - 7.2 - Fix escaping of BLOB/RAW to bytea data import causing import to crash. Thanks to Ozmen Emre Demirkol for the report. - Add support to default value into functions parameter (PG >= 8.4). Can be activated with a new configuration directive: PG_SUPPORTS_DEFAULT. Default is 1, activated. - Fix bad ending of exported function: remove trailing chars after END. - Add support to WHEN clause on triggers (PG >= 9.0), can be activated with a new configuration directive: PG_SUPPORTS_WHEN. - Add support to INSTEAD OF usage on triggers (incoming PG >= 9.1). Can be activated with a new configuration directive: PG_SUPPORTS_INSTEADOF. - Fix error using SKIP directive. Thanks to Laurent Renard from Cap Gemini for the report. - Fix missing perl object instance in format_data() function. - Fix duplicate procedure or function when export type use both FUNCTION and PROCEDURE. 2010 12 04 - 7.1 - Improve direct DBD::Pg data export/import speed by 10. - Add --section=3 in pod2man call into Makefile.PL. Thanks to Julian Moreno Patino for the report. - Renamed ChangeLog into changelog to avoid upstream warning with Debian package. Thanks to Julian Moreno Patino for the suggestion. - Fix some spelling mistakes in doc/Ora2Pg.pod. Thanks to Julian Moreno Patino for the fix. - Fix release version into Ora2Pg.pm and PLSQL.pm, was still in 6.5. - Fix direct data export/import using DBD::Pg. Thanks to Laurent Renard from Cap Gemini for the report. - Fix drop/create contraints and index during direct data export/import using DBD::Pg. Thanks to Thierry Grasland from Cap Gemini for the report. 2010 11 23 - 7.0 - Rename ora2pg perl script into ora2pg.pl in sources because Windows users can't extract the tarball. During install it is renamed into ora2pg. Thanks to Andrew Marlow for the report. - Fix doinst.sh for SlackWare Slackbuid packaging. - The DEFER_FKEY configuration directive has been fixed as it only works in a transaction. Note that foreign keys must have been created as DEFERRABLE or it also will not works. Thanks to Igor Gelman for the report. - Add DROP_FKEY configuration directive to force deletion of foreign keys before the import and recreate them and the end of the import. This may help if DEFER_FKEY not works for you. - Add DROP_INDEX configuration directive to force deletion of all indexes except the automatic index (primary keys) before data import and to recreate them at end. This can be used to gain speed during import. - Add TSMSYS, FLOWS_020100 and FLOWS_FILES to the owners exclude list. This concern the SRS$ table and all tables begining with 'WWV_FLOW_' - Change the way DATA_LIMIT is working. It must be used now to set the bulk size of tuples return at once. Default is 10000. - Improve data export speed by 6! The data export code has been entierly rewritten and the speed gain is really fun. - Add OUTPUT_DIR configuration directive to set a base directory where all dumped files must be written. Default: current directory. - Change value of default numeric(x) type from float to bigint and change default numeric(x,y) type to double precision. - Change conversion type for BFILE from text to bytea. 2010 09 10 - 6.4 - Configuration directives SHOWTABLEID, MIN and MAX are now obsolete and has been definitively removed. They were never used and add too much confusion. - Fix bug in column name replacement where table name was also replaced. Thank to Jean-Paul Argudo from DALIBO for the report. - Fix case sensitive errata in PG schema search path. Thank to Jean-Paul Argudo from DALIBO for the report. - Remove double \n at end of debug message. - Fix debug mode not activated if the DEBUG directive is enable and the -d command line is not present. - Add unbuffered output for debug message. UPGRADE: simply override the Ora2Pg.pm Perl module where it is installed. 2010 07 22 - 6.3 - Fix Oracle 8i compatibility error during schema extraction complaining that column CHAR_LENGTH doesn't exist. Thanks to Philippe Rimbault for the report. Note that the error message is still displayed but tagged as WARNING only. - Fix error using the IMPORT option on a read_conf method call. Thanks to Diogo Biazus for the report. - Fix export of sequences that does not handle maxvalue well and can be lower than minvalue. Thanks to Nathalie Doremieux for the report. UPGRADE: Just override Ora2Pg.pm 2010 06 15 - 6.2 - Change default transaction isolation level from READ ONLY to SERIALIZABLE to ensure consistency during data export. Thanks to Hans-Jurgen Schonig from postgresql-support.de - Add the TRANSACTION configuration directive to allow overriding of the isolation level. Value can be readonly, readwrite, committed and serializable. The last is the default. 2010 05 07 - 6.1 - Fix error on partition export following schema definition. - Add first support to export Oracle user defined types. - Add CTXSYS,XDB,WMSYS,SYSMAN,SQLTXPLAIN,MDSYS,EXFSYS,ORDSYS,DMSYS, OLAPSYS to the sysuser default exclusion list. - PLSQL.pm: Add automatic translation of Oracle raise_application_error and dup_val_on_index to PG RAISE EXCEPTION and UNIQUE_VIOLATION. - Change/fallback to a lower case package name (ora2pg-6.x.tar.gz). - Change default convert type for 'LONG RAW' to bytea. - Add PG_SCHEMA configuration directive to defined a coma delimited list of schema to use in SET search_path PostgreSQL command. 2010 02 28 - 6.0 - Publish a dedicated site to Ora2Pg at http://ora2pg.darold.net/ - Add export of Oracle table partitoning. See export type PARTITION. - Add command line arguments to perl script ora2pg. See --help for a full listing of these option. The most interesting is --type to change the export type directly at command execution without needing to edit the configuration file, --plsql to directly enable PLSQL to PLPSQL code conversion and --source, --user --password to set Oracle data source. There's also --namespace to set the Oracle schema. - Create all file for standard Perl module install. Install is now done with: perl Makefile.PL && make && make install - Move Ora2Pg license from Perl Artistics to GPLv3. - Move PLSQL package as Ora2Pg::PLSQL for standard Perl module install. - Remove use of Perl module String::Random. - Rename program ora2pg.pl into ora2pg for standard usage. - Fix extra double quote on column name of index export. Thanks to Guillaume Lelarge for the patch. - Add packaging facilities to build RPM, SlackBuild and Debian packages. - Fix miss handling of Ora2Pg.pm options at object instance init. - Configuration file ora2pg.conf is now generated by Makefile.PL 2009 12 18 - 5.5 - Fix CONSTANT declaration in Procedure/Function/Package export. - Fix length of char and varchar on multibyte like UTF8 encoding. Thanks to Ali Pouya for the patch. - Fix view export where alias to column in Oracle not use 'AS' and PostgreSQL required it. Thanks to Ali Pouya for the report. - Add type replacement of sql variable in PLSQL code (PLSQL.pm). Thanks to vijay for the patch. 2009 07 15 - 5.4 - Fix bug introduced in multiple output file feature. This bug force Ora2pg to crach after the first table export when output is wanted in a single file. Thanks to Leo Mannhart for the report. - Fix debug filename output on multiple export file. Thanks to Leo Mannhart for the report. 2009 07 07 - version 5.3 - Fix wrong escaping of data named as column during view export. Thank to Andrea Agosti for the patch. - Allow export of datas into one file per table. See FILE_PER_TABLE configuration directive. Thanks to Alexandre - Aldeia Digital for the idea. 2009 06 19 - version 5.2 - Fix order of the column name of the view which was not preserved. Now ordered by COLUMN_ID. Thank to Andrea Agosti for the report. - Fix case sensitivity in VIEW extraction. Thank to Andrea Agosti for the patch. 2009 03 06 - version 5.1 - Fix missing -U username at Pg connection. Thanks to Brendan Richards. - Fix $ENV{ORACLE_HOME} and $ENV{NLS_LANG} to not being overwritten by configuration settings if they are already defined in environment. - Fix typo in ora2pg.pl where keep_pkey_names was replaced by keep_pkeys_name and so prevent use of KEEP_PKEY_NAMES in configuration. Thanks to Olivier Mazain for the report. - Configuration file directives are now case insensitive. - Force $type parameter given to _sql_type() to be uppercase in that methode instead of during function call. Thanks to Ali Pouya for the report. - Modify ora2pg.pl to remove the obsolete call to export_data(). Use only export_schema() now. - Modify ora2pg.pl to simplify it. Reading configuration is now done internally by Ora2Pg.pm as well as all other initialization process. You can always overwrite all configuration options into call to new Ora2Pg(). Now ora2pg.pl can be as simple as: use Ora2Pg; my $schema = new Ora2Pg('config' => "/etc/ora2pg.conf"); $schema->export_schema(); exit(0); This will be less confusing. You can upgrade Ora2Pg.pm without carring about that, backward compatibility with previous version is preserved. - Review entire documentation with the great help of Ali Pouya. - Add type BOOLEAN converted to boolean. - PG_SUPPORTS_INOUT is now enabled by default in the configuration file - SQL and PL/SQL to PLPGSQL converter: .Replace MINUS call to EXCEPT .Replace DBMS_OUTPUT.put, DBMS_OUTPUT.put_line, DBMS_OUTPUT.new_line by the PLPGSQL equivalent: RAISE NOTICE .Rewrite function/procedure/package convertion functions. This Oracle SQL converter for function/procedure/package is now only applied if the configuration directive PLSQL_PGSQL is enable, else these Oracle code are exported as is. Thanks to Ali Pouya for the help. - Reserved call to sql transaction only for DATA export type. Others export type now use \set ON_ERROR_STOP ON. Thanks to Ali Pouya. - Fix tablespace creation into schema (missing search_path). Thanks to Olivier Mazain. - Fix the type returned by the _sql_type() method in the case of a numeric with null len and pg_numeric_type is set. Thanks to Ali Pouya. - Change function body delimiter to $body$ to allow use of $$ into the body as quote replacement. Thanks to Ali Pouya. - Fix returns type from function. If multiple OUT parameters: RECORD, if only one OUT parameter, return his type. If no out parameter: return VOID. Thanks to Ali Pouya. - Fix export DATA when the name of a column in the table match COMMENT, AUDIT or any other defined reserved words. These reserved words are defined in a new configuration variable ORA_RESERVED_WORDS. It accept a list of comma separated reserved words. Thanks to Andrea Agosti for the report. - Fix configuration parser that omit custom SYSUSERS definition. 2009 02 13 - version 5.0 - Fix places where $self->{prefix} where not used. This prefix is used to replace DBA_... objects into ALL_... objects. Thanks to Daniel Scott report and patch. - Fix some problem on trigger export (missing ending semicolon, return opaque replaced by return trigger, add missing return new value, single quote for delimitating the function body hits against quotes inside the function). Thanks to Luca DallOlio for reports and patches. - Add first attempt to rewrite plsql code to plpgsql code (see function plsql_to_plpgsql in new perl module PLSQL.pm). There's a configuration option named PLSQL_PGSQL to activate the convertion. 2008 12 16 - version 4.11 - Fix Ora2Pg failure on Oracle database with case sensitive tables. Thanks to Marc Cousin for report and patch. - Fix missing schema name in query when extract views as tables. 2008 12 04 - version 4.10 - Fix missing replacement of table name on disable triggers when required. - Fix some malformed debug output messages. - Add the capability to extract data from view as if it was a table. This is usefull if you want to export/import data from an Oracle view into a Pg table. There's nothing special to do, just to give the view name into the TABLES configuration directive and set TYPE to DATA or COPY. If views are not specified in the TABLES directive there's not view export but only table data. - Add capability to extract views structure as table schema. There's nothing special to do, just to give the view name into the TABLES configuration directive and set TYPE to TABLE. This will not extract constraints or other table tuning from table used in the view. Thanks to Groupe SAMSE for the feature request. 2008 10 27 - version 4.9 - Modify the DISABLE_TABLE_TRIGGERS configuration option. Should be now replaced by DISABLE_TRIGGERS, but compatibility is preserved. - Add DISABLE_SEQUENCE configuration option to not export alter sequence after COPY or DATA export. - Fix extraction of function based index that appears as SYS_NC.... Thanks to Bozkurt Erkut from SONY for the report 2008 09 04 - version 4.8 - Add SYSUSERS configuration option that allow you to specify a coma separated list of Oracle System user/schema to exclude from extracted object. By default it only exclude user SYS,SYSTEM,DBSNMP,OUTLN and PERFSTAT - Add support to other binary mode output than ':raw' to avoid the Perl error message:"Wide character in print". See the BINMODE configuration directive. This will help a lot if you have UTF-8 records. Thank to Guillaume Demillecamps for the report. - Fix double escaping of special character. Thank to Guillaume Demillecamps for the report. 2008 01 25 - version 4.7 - Add support to regular expressions in the exclusion list. Thanks to Peter Eisentraut - Fix misformatted SQL string in function _extract_sequence_info. Thanks to Bernd Helmle. - Add escaping of backslash on COPY output. Thanks to Peter Eisentraut 2008 01 03 - version 4.6 - Applied a patch to add ALTER SEQUENCE statements to the dump to adjust the sequence to the correct values after DATA and COPY dumps. Thanks to Bernd Helmle. - Applied a patch which fixes problems with broken COPY output when extracting data from Orace databases with embedded tabs, carriage returns and line feeds. Thanks to Bernd Helmle. - Move the project to PgFoundry 2007 06 20 - version 4.5 - Fix columns order in index extraction. Thanks to Ugo Brunel from BULL. 2007 04 30 - version 4.4 - Fix missing single quote in role extraction. - Add configuration directive NOESCAPE to enable/disable escaping characters during data extraction. Default is enabled. - Add TIMESTAMP, BINARY_FLOAT and BINARY_DOUBLE data type translation. - Add DATA_TYPE configuration directive to allow user defined data type translation. - Add NLS_LANG configuration directive to set Oracle database encoding and enforce a default language-setting in ora2pg.pl. Thanks to Lars Weber 2007 04 03 - version 4.3 - Fix duplicate view export. Add schema selector to views. Thank to Ugo BRUNEL from BULL for the fix. - Remove 'use strict' to prevent failure on certain condition. Thank to Andrea Schnabl for the report. 2006 06 08 - version 4.2 - Fix a miss taping on constraint type search that convert unique key to primary key. Thank to Ugo BRUNEL (BULL) for the patch. - Fix case sensitivity on CHECK constraint that could cause problem when check value is uppercase. Thank to Ugo BRUNEL (BULL) for the patch. 2006 03 28 - version 4.1 - Fix a problem when using data_limit and where clause. Thank to Rene Bentzen for the patch. - Add enable/disable trigger on data import. Thank to Bernd Helmle. - Fix escaping of chr(13) MS crashing data import into PG. Thank to Ugo Brunel (BULL). 2006 03 22 - version 4.0 - Add validation of the requested schema in the database before all. Thanks to Max Walton for the idea. - Add multiple export type at the same time. Thanks to Max Walton for the idea. - Add support for in/out/inout function parameter. See PG_SUPPORTS_INOUT configuration option. Thanks to Bernd Helmle for this great contribution/patch. - Add support for ROLES with Pg v8.1+. See PG_SUPPORTS_ROLE configure option. 2006 02 10 - version 3.4 This release add better support to Oracle grant, function and grant extraction. Great thanks to the Pg team! - Add preservation of oracle primary key names. See KEEP_PKEY_NAMES configuration option. Thanks to Antonios Christofides for this patch. - Fix bug in case insensitive check constrainte. Thanks to Wojciech Szenajch for the patch. - Fix saving data to files correctly (binmod) when the oracle database contains utf8 chars. Thanks to Richard Chen for the report. - Fix bug on view extraction when a column contains the word WITH. Thanks to Richard Chen for the patch. - Fix wrong mapping between tge data type in Oracle "number(10)" and Postgresql, which should be "integer" and not "bigint". Thanks to Sergio Freire for the patch. - Fix bug in EXCLUDE configuration directive parsing. Thanks to Matt Miller for the patch. 2005 02 22 - version 3.3 - Fix bug "Modification of a read-only value attempted" 2005 02 11 - version 3.2 - Fix patch error on column position sort - Replace 'now' by CURRENT_TIMESTAMP on SYSDATE replacement - Fix bytea type that was not quoted. 2005 02 10 - version 3.1 - Fix bug on deferrable constraint. Thanks to Antonios Christofide for the patch. - Fix problem on defer_fkey that should be in a transaction. Thanks to Antonios Christofide for the patch. - Add sort by column position during schema extraction. - Add support to SYSDATE. Thanks to David Cotter-Alatto Technologies Ltd 2004 12 24 - version 3.0 - Add 'TABLESPACE' extraction type to create PostgreSQL v8 tablespace. 2004 12 24 - version 2.9 - Debuging output rewrite. Thanks to Antonios Christofide for help. - Add 'PG_NUMERIC_TYPE' configuration option to replace portable numeric type into PostgreSQL internal type (smallint, integer, bigint, real and float). 2004 12 24 - version 2.8 - Fix/add support to data export of type BLOB, RAW and LONG RAW. Thanks to Antonios Christofide for help. 2004 12 23 - version 2.7 - Add 'FKEY_DEFERRABLE' configuration option to force foreign key constraints to be exported as deferrable. Thanks to Antonios Christofide for help. - Add 'DEFER_FKEY' configuration option to defer all foreign key constraints during data export. Thanks to Antonios Christofide for help. 2004 12 23 - version 2.6 - Fix duplicate output during export. Thanks to Adriano Bonat for the report. - Fix data limit infinite loop during data extraction. Thanks to Thomas REISS for the report. - Add 'GEN_USER_PWD' configuration option allowing to generate a random password. Thanks to Antonios Christofide for help. (Require String::Random from CPAN). - Fix USER/ROLES/GRANT extraction problem. Now all users are dumped. All roles are translated to PostgreSQL groups. All grants are exported. YOU MUST EDIT the output file to rewrite real privilege and match your needs. Thanks to Antonios Christofide for help. - Fix split COPY export into multiple transaction for large data export. The number of row per transaction is set to 'DATA_LIMIT' value. A value of O mean all in a single transaction. 2004 10 13 - version 2.5 - Fix extraction problem when the connection to Oracle DB is not as DBA. 2004 08 22 - version 2.4 - Fix bug in DBI errstr call. - Add CASE_SENSITIVE configuration option to allow case sensitivity on Add a new configuration directive 'USER_GRANTS' to do that. Thanks to Octavi Fors for the report. object name. Thanks to Thomas Wegner. - Fix major bug in unique keys extraction. Thanks to Andreas Haumer and Marco Lombardo for their great help. - Add CHECK constraint extration. Thanks again to Andreas Haumer. - Add IMPORT configuration option to include common configuration file throught multiple configuration files.Thanks to Adam Sah and Zedo Inc. - Add SKIP configuration option to turning off extraction of certain - schema features. Thanks to Adam Sah and Zedo Inc. - Fix bug in excluded tables - Fix backslash escaping. Thanks to Adam Sah and Zedo Inc. - Add REPLACE_TABLES configuration option to change table name during data extraction. - Add REPLACE_COLS configuration option to change columns name during data extraction. - Add WHERE configuration option to add where clause to each table or specific tables during extraction. Usefull for replication. Thanks to Adam Sah and Zedo Inc. - Add progress indicators (per 1000 rows) and performance results during data extraction in debug mod. Thanks to Adam Sah and Zedo Inc. - Add Gzip and Bzip2 compress to output file if extension .gz or .bz2. Gzip compress require perl module Compress::Zlib from CPAN. Thanks to Adam Sah for the idea. 2004 04 13 - Version 2.3 - Fix bug in date/time conversion when using data export limit. Thanks to Andreas Haumer. - Add sort order when extracting tables and data to respect the TABLES limited extraction array write order. Usefull if you have foreign key constraints. Thanks to Andreas Haumer for the idea. 2004 04 13 - Version 2.2 - Add EXCLUDE configuration option to allow table exclusion from all extraction. - Fix a bug in escaping single quote on data export. 2004 03 09 - Version 2.1 - Fix COPY output by replacing special character. - Add configuration file usefull for people who don't have Perl in mind Thank's to Tanya Krasnokutsky to force me to do that :-) - Fix other minor problem. 2002 12 26 - Version 2.0 - Clean code. - Fix COPY output on column value with EOL and add column naming. - Add support to the PostgreSQL 7.3 schema. So Oracle schema can now be exported. (see export_schema init option) - Remove data extraction limit (old default: 10) so each tuple will be dump by default. 2002 12 03 - Version 1.12 I have fixed 2 bugs when using it against Oracle 817R3 on linux. - Fix problem regarding RI constraints, the owner name was not getting into the sql statement. Thank to Ian Boston. - Moved all the RI constraints out of the create table statement. Thank to Ian Boston for this contribution. This was a major request from Ora2pg users. 2002 09 27 - Version 1.11 - Fix a problem when retrieving package+package body. Thanks to Mike WILHELM-HILTZ. - Set LongReadLen to 100000 when exporting table information. Many users reports this kind of error: A-01406 LongReadLen too small and/or LongTruncOk not set. This should fix the problem else you must increase the value. - Filtering by owner for better performance when retreiving database schema. Thanks to Jefferson MEDEIROS. 2002 07 29 - Version 1.10 - Fix a problem with local settings regarding decimal separator (all , are changed to .) Thank to Jan Kester. 2002 06 04 - Version 1.9 - Fix a problem on exporting data which fill NULL instead of 0 or empty string. Thanks to Jan Kester. - Add time + date when export data [ tochar('YYYY-MM-DD HH24:MI:SS') ]. Thanks to Paolo Mattioli. 2002 03 05 - Version 1.8 - Add Oracle type FLOAT conversion to float8. - Add column alias extraction on view. Thanks to Jean-Francois RIPOUTEAU - Add PACKAGE extraction (type => DATA). 2002 02 14 - Version 1.7 - Remove export of OUTLINE object type. Thanks to Jean-Paul ARGUDO. 2002 01 07 - Version 1.6 - Fix problem exporting NULL value. Thanks to Stephane Schildknecht. 2001 12 28 - Version 1.5 - Fix LongReadLen problem when exporting Oracle data on LONG and LOB types. Thanks to Stephane Schildknecht for report and test. - Add more precision on NUMBER type conversion - Add conversion of type LONG, LOB, FILE - Fix a problem when extracting data, sometime table could need to be prefixed by the schema name. - Fix output of Oracle data extraction. It now require a call to function export_data(). 2001 06 27 - Version 1.4 - Add online Oracle data extraction and insertion into PG database. - Data export as insert statement (type => DATA) - Data export as copy from stdin statement (type => COPY) 2001 06 20 - Version 1.3 - Grant/privilege extraction are now done separatly with option type=>'GRANT' - Sequence extraction with the option type=>'SEQUENCE' - Trigger extraction with the option type=>'TRIGGER' - Function extraction with the option type=>'FUNCTION' and type=>'PROCEDURE' - Complete rewrite of the foreign key extraction - Fix incorrect type translation and many other bug fix - Add schema only extraction by option schema => 'MYSCHEM' 2001 05 11 - Version 1.2 - Views extraction is now really done with the option type=>'VIEW' - Add indexes extraction on tables. - Changes name of constraints, default is now used. - Add debug printing to see that the process is running :-) - Add extraction of only required tablename. - Add extraction of only n to n table indice. Indices of extraction can be obtained with the option showtableid set to 1. - Fix print of NOT NULL field. - Complete rewrite of the grant extraction - Complete rewrite of most things 2001 05 09 - Version 1.1 - Add table grant extraction based on group. Oracle ROLES are exported as groups in PG 2001 05 09 - Initial version 1.0 ------------------------------------------------------------------------------ Special thanks to Ali Pouya for documentation review. All my recognition to Ali Pouya and Olivier Mazain for their great work in the package and function export. Thanks to Jean-Paul Argudo for the time spent to heavily testing Ora2Pg. Special thanks to Josian Larcheveque and Stephane Silly as Oracle DBA and their "patience". Special Thanks to Dominique Legendre for his help on Spatial support and all the tests performed. Many thanks for all congratulation message, idea and bug report+fix I received. Very special thanks to Jean-Paul Argudo that represent Ora2Pg at Linux Solution Paris 2005. Gilles DAROLD ora2pg-25.0/doc/000077500000000000000000000000001500113072400133705ustar00rootroot00000000000000ora2pg-25.0/doc/Ora2Pg.pod000066400000000000000000004511241500113072400151750ustar00rootroot00000000000000=head1 NAME Ora2Pg - Oracle to PostgreSQL database schema converter =head1 DESCRIPTION Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scans it automatically and extracts its structure or data, then generates SQL scripts that you can load into your PostgreSQL database. Ora2Pg can be used for anything from reverse engineering Oracle database to huge enterprise database migration or simply replicating some Oracle data into a PostgreSQL database. It is really easy to use and doesn't require any Oracle database knowledge other than providing the parameters needed to connect to the Oracle database. =head1 FEATURES Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), the only thing you have to modify is the configuration file ora2pg.conf by setting the DSN to the Oracle database and optionally the name of a schema. Once that's done you just have to set the type of export you want: TABLE with constraints, VIEW, MVIEW, TABLESPACE, SEQUENCE, INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, PACKAGE, PARTITION, TYPE, INSERT or COPY, FDW, QUERY, KETTLE, SYNONYM. By default Ora2Pg exports to a file that you can load into PostgreSQL with the psql client, but you can also import directly into a PostgreSQL database by setting its DSN into the configuration file. With all configuration options of ora2pg.conf you have full control of what should be exported and how. Features included: - Export full database schema (tables, views, sequences, indexes), with unique, primary, foreign key and check constraints. - Export grants/privileges for users and groups. - Export range/list partitions and sub partitions. - Export a table selection (by specifying the table names). - Export Oracle schema to a PostgreSQL 8.4+ schema. - Export predefined functions, triggers, procedures, packages and package bodies. - Export full data or following a WHERE clause. - Full support of Oracle BLOB object as PG BYTEA. - Export Oracle views as PG tables. - Export Oracle user defined types. - Provide some basic automatic conversion of PLSQL code to PLPGSQL. - Works on any platform. - Export Oracle tables as foreign data wrapper tables. - Export materialized view. - Show a report of an Oracle database content. - Migration cost assessment of an Oracle database. - Migration difficulty level assessment of an Oracle database. - Migration cost assessment of PL/SQL code from a file. - Migration cost assessment of Oracle SQL queries stored in a file. - Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) - Export Oracle locator and spatial geometries into PostGis. - Export DBLINK as Oracle FDW. - Export SYNONYMS as views. - Export DIRECTORY as external table or directory for external_file extension. - Dispatch a list of SQL orders over multiple PostgreSQL connections - Perform a diff between Oracle and PostgreSQL database for test purpose. - MySQL/MariaDB and Microsoft SQL Server migration. Ora2Pg does its best to automatically convert your Oracle database to PostgreSQL but there's still manual works to do. The Oracle specific PL/SQL code generated for functions, procedures, packages and triggers has to be reviewed to match the PostgreSQL syntax. You will find some useful recommendations on porting Oracle PL/SQL code to PostgreSQL PL/PGSQL at "Converting from other Databases to PostgreSQL", section: Oracle (http://wiki.postgresql.org/wiki/Main_Page). See http://ora2pg.darold.net/report.html for a HTML sample of an Oracle database migration report. =head1 INSTALLATION All Perl modules can always be found at CPAN (http://search.cpan.org/). Just type the full name of the module (ex: DBD::Oracle) into the search input box, it will brings you the page for download. Releases of Ora2Pg stay at SF.net (https://sourceforge.net/projects/ora2pg/). Under Windows you should install Strawberry Perl (http://strawberryperl.com/) and the OSes corresponding Oracle clients. Since version 5.32 this Perl distribution include pre-compiled driver of DBD::Oracle and DBD::Pg. =head2 Requirement The Oracle Instant Client or a full Oracle installation must be installed on the system. You can download the RPM from Oracle download center: rpm -ivh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm rpm -ivh oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm or simply download the corresponding ZIP archives from Oracle download center and install them where you want, for example: /opt/oracle/instantclient_12_2/ You also need a modern Perl distribution (perl 5.10 and more). To connect to a database and proceed to his migration you need the DBI Perl module > 1.614. To migrate an Oracle database you need the DBD::Oracle Perl modules to be installed. To install DBD::Oracle and have it working you need to have the Oracle client libraries installed and the ORACLE_HOME environment variable must be defined. If you plan to export a MySQL database you need to install the Perl module DBD::MySQL which requires that the mysql client libraries are installed. If you plan to export a SQL Server database you need to install the Perl module DBD::ODBC which requires that the unixODBC package is installed. On some Perl distribution you may need to install the Time::HiRes Perl module. If your distribution doesn't include these Perl modules you can install them using CPAN: perl -MCPAN -e 'install DBD::Oracle' perl -MCPAN -e 'install DBD::MySQL' perl -MCPAN -e 'install DBD::ODBC' perl -MCPAN -e 'install Time::HiRes' otherwise use the packages provided by your distribution. =head2 Optional By default Ora2Pg dumps export to flat files, to load them into your PostgreSQL database you need the PostgreSQL client (psql). If you don't have it on the host running Ora2Pg you can always transfer these files to a host with the psql client installed. If you prefer to load export 'on the fly', the perl module DBD::Pg is required. Ora2Pg allows you to dump all output in a compressed gzip file, to do that you need the Compress::Zlib Perl module or if you prefer using bzip2 compression, the program bzip2 must be available in your PATH. If your distribution doesn't include these Perl modules you can install them using CPAN: perl -MCPAN -e 'install DBD::Pg' perl -MCPAN -e 'install Compress::Zlib' otherwise use the packages provided by your distribution. =head2 Instruction for SQL Server For SQL Server you need to install the unixodbc package and the Perl DBD::ODBC driver: sudo apt install unixodbc sudo apt install libdbd-odbc-perl or sudo yum install unixodbc sudo yum install perl-DBD-ODBC sudo yum install perl-DBD-Pg then install the Microsoft ODBC Driver for SQL Server. Follow the instructions relative to your operating system from here: https://docs.microsoft.com/fr-fr/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver16 Once it is done set the following in the /etc/odbcinst.ini file by adjusting the SQL Server ODBC driver version: [msodbcsql18] Description=Microsoft ODBC Driver 18 for SQL Server Driver=/opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.0.so.1.1 UsageCount=1 See ORACLE_DSN to know how to use the driver to connect to your MSSQL database. =head2 Installing Ora2Pg Like any other Perl Module Ora2Pg can be installed with the following commands: tar xjf ora2pg-x.x.tar.bz2 cd ora2pg-x.x/ perl Makefile.PL make && make install This will install Ora2Pg.pm into your site Perl repository, ora2pg into /usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. On Windows(tm) OSes you may use instead: perl Makefile.PL gmake && gmake install This will install scripts and libraries into your Perl site installation directory and the ora2pg.conf file as well as all documentation files into C:\ora2pg\ To install ora2pg in a different directory than the default one, simply use this command: perl Makefile.PL PREFIX= make && make install then set PERL5LIB to the path to your installation directory before using Ora2Pg. export PERL5LIB= ora2pg -c config/ora2pg.conf -t TABLE -b outdir/ =head2 Packaging If you want to build the binary package for your preferred Linux distribution take a look at the packaging/ directory of the source tarball. There is everything to build RPM, Slackware and Debian packages. See README file in that directory. =head2 Installing DBD::Oracle Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle database from perl DBI. To get DBD::Oracle get it from CPAN a perl module repository. After setting ORACLE_HOME and LD_LIBRARY_PATH environment variables as root user, install DBD::Oracle. Proceed as follow: export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib export ORACLE_HOME=/usr/lib/oracle/12.2/client64 perl -MCPAN -e 'install DBD::Oracle' If you are running for the first time it will ask many questions; you can keep defaults by pressing ENTER key, but you need to give one appropriate mirror site for CPAN to download the modules. Install through CPAN manually if the above doesn't work: #perl -MCPAN -e shell cpan> get DBD::Oracle cpan> quit cd ~/.cpan/build/DBD-Oracle* export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib export ORACLE_HOME=/usr/lib/oracle/11.2/client64 perl Makefile.PL make make install Installing DBD::Oracle require that the three Oracle packages: instant-client, SDK and SQLplus are installed as well as the libaio1 library. If you are using Instant Client from ZIP archives, the LD_LIBRARY_PATH and ORACLE_HOME will be the same and must be set to the directory where you have installed the files. For example: /opt/oracle/instantclient_12_2/ =head1 CONFIGURATION Ora2Pg configuration can be as simple as choosing the Oracle database to export and choose the export type. This can be done in a minute. By reading this documentation you will also be able to: - Select only certain tables and/or column for export. - Rename some tables and/or column during export. - Select data to export following a WHERE clause per table. - Delay database constraints during data loading. - Compress exported data to save disk space. - and much more. The full control of the Oracle database migration is taken though a single configuration file named ora2pg.conf. The format of this file consist in a directive name in upper case followed by tab character and a value. Comments are lines beginning with a #. There's no specific order to place the configuration directives, they are set at the time they are read in the configuration file. For configuration directives that just take a single value, you can use them multiple time in the configuration file but only the last occurrence found in the file will be used. For configuration directives that allow a list of value, you can use it multiple time, the values will be appended to the list. If you use the IMPORT directive to load a custom configuration file, directives defined in this file will be stores from the place the IMPORT directive is found, so it is better to put it at the end of the configuration file. Values set in command line options will override values from the configuration file. =head2 Ora2Pg usage First of all be sure that libraries and binaries path include the Oracle Instant Client installation: export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration file, if the file exist you can simply execute: /usr/local/bin/ora2pg or under Windows(tm) run ora2pg.bat file, located in your perl bin directory. Windows(tm) users may also find a template configuration file in C:\ora2pg If you want to call another configuration file, just give the path as command line argument: /usr/local/bin/ora2pg -c /etc/ora2pg/new_ora2pg.conf Here are all command line parameters available when using ora2pg: Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] -a | --allow str : Comma separated list of objects to allow from export. Can be used with SHOW_COLUMN too. -b | --basedir dir: Set the default output directory, where files resulting from exports will be stored. -c | --conf file : Set an alternate configuration file other than the default /etc/ora2pg/ora2pg.conf. -C | --cdc_file file: File used to store/read SCN per table during export. default: TABLES_SCN.log in the current directory. This is the file written by the --cdc_ready option. -d | --debug : Enable verbose output. -D | --data_type str : Allow custom type replacement at command line. -e | --exclude str: Comma separated list of objects to exclude from export. Can be used with SHOW_COLUMN too. -h | --help : Print this short help. -g | --grant_object type : Extract privilege from the given object type. See possible values with GRANT_OBJECT configuration. -i | --input file : File containing Oracle PL/SQL code to convert with no Oracle database connection initiated. -j | --jobs num : Number of parallel process to send data to PostgreSQL. -J | --copies num : Number of parallel connections to extract data from Oracle. -l | --log file : Set a log file. Default is stdout. -L | --limit num : Number of tuples extracted from Oracle and stored in memory before writing, default: 10000. -m | --mysql : Export a MySQL database instead of an Oracle schema. -M | --mssql : Export a Microsoft SQL Server database. -n | --namespace schema : Set the Oracle schema to extract from. -N | --pg_schema schema : Set PostgreSQL's search_path. -o | --out file : Set the path to the output file where SQL will be written. Default: output.sql in running directory. -O | --options : Used to override any configuration parameter, it can be used multiple time. Syntax: -O "PARAM_NAME=value" -p | --plsql : Enable PLSQL to PLPGSQL code conversion. -P | --parallel num: Number of parallel tables to extract at the same time. -q | --quiet : Disable progress bar. -r | --relative : use \ir instead of \i in the psql scripts generated. -s | --source DSN : Allow to set the Oracle DBI datasource. -S | --scn SCN : Allow to set the Oracle System Change Number (SCN) to use to export data. It will be used in the WHERE clause to get the data. It is used with action COPY or INSERT. -t | --type export: Set the export type. It will override the one given in the configuration file (TYPE). -T | --temp_dir dir: Set a distinct temporary directory when two or more ora2pg are run in parallel. -u | --user name : Set the Oracle database connection user. ORA2PG_USER environment variable can be used instead. -v | --version : Show Ora2Pg Version and exit. -w | --password pwd : Set the password of the Oracle database user. ORA2PG_PASSWD environment variable can be used instead. -W | --where clause : Set the WHERE clause to apply to the Oracle query to retrieve data. Can be used multiple time. --forceowner : Force ora2pg to set tables and sequences owner like in Oracle database. If the value is set to a username this one will be used as the objects owner. By default it's the user used to connect to the Pg database that will be the owner. --nls_lang code: Set the Oracle NLS_LANG client encoding. --client_encoding code: Set the PostgreSQL client encoding. --view_as_table str: Comma separated list of views to export as table. --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT --cost_unit_value minutes: Number of minutes for a cost evaluation unit. default: 5 minutes, corresponds to a migration conducted by a PostgreSQL expert. Set it to 10 if this is your first migration. --dump_as_html : Force ora2pg to dump report in HTML, used only with SHOW_REPORT. Default is to dump report as simple text. --dump_as_csv : As above but force ora2pg to dump report in CSV. --dump_as_json : As above but force ora2pg to dump report in JSON. --dump_as_sheet : Report migration assessment with one CSV line per database. --init_project name: Initialise a typical ora2pg project tree. Top directory dump_as_* selected switches, suffixes will be .html, .csv, .json. --init_project name: Initialise a typical ora2pg project tree. Top directory will be created under project base dir. --project_base dir : Define the base dir for ora2pg project trees. Default is current directory. --print_header : Used with --dump_as_sheet to print the CSV header especially for the first run of ora2pg. --human_days_limit num : Set the number of person-days limit where the migration assessment level switch from B to C. Default is set to 5 person-days. --audit_user list : Comma separated list of usernames to filter queries in the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT and QUERY export type. --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. --pg_user name : Set the PostgreSQL user to use. --pg_pwd password : Set the PostgreSQL password to use. --count_rows : Force ora2pg to perform a real row count in TEST, TEST_COUNT and SHOW_TABLE actions. --no_header : Do not append Ora2Pg header to output file --oracle_speed : Use to know at which speed Oracle is able to send data. No data will be processed or written. --ora2pg_speed : Use to know at which speed Ora2Pg is able to send transformed data. Nothing will be written. --blob_to_lo : export BLOB as large objects, can only be used with action SHOW_COLUMN, TABLE and INSERT. --cdc_ready : use current SCN per table to export data and register them into a file named TABLES_SCN.log per default. It can be changed using -C | --cdc_file. --lo_import : use psql \lo_import command to import BLOB as large object. Can be use to import data with COPY and import large object manually in a second pass. It is recquired for BLOB > 1GB. See documentation for more explanation. --mview_as_table str: Comma separated list of materialized views to export as regular table. --drop_if_exists : Drop the object before creation if it exists. --delete clause : Set the DELETE clause to apply to the Oracle query to be applied before importing data. Can be used multiple time. --oracle_fdw_prefetch: Set the oracle_fdw prefetch value. Larger values generally result in faster data transfer at the cost of greater memory utilisation at the destination. See full documentation at https://ora2pg.darold.net/ for more help or see manpage with 'man ora2pg'. ora2pg will return 0 on success, 1 on error. It will return 2 when a child process has been interrupted and you've gotten the warning message: "WARNING: an error occurs during data export. Please check what's happen." Most of the time this is an OOM issue, first try reducing DATA_LIMIT value. For developers, it is possible to add your own custom option(s) in the Perl script ora2pg as any configuration directive from ora2pg.conf can be passed in lower case to the new Ora2Pg object instance. See ora2pg code on how to add your own option. Note that performance might be improved by updating stats on oracle: BEGIN DBMS_STATS.GATHER_SCHEMA_STATS DBMS_STATS.GATHER_DATABASE_STATS DBMS_STATS.GATHER_DICTIONARY_STATS END; =head2 Generate a migration template The two options --project_base and --init_project when used indicate to ora2pg that he has to create a project template with a work tree, a configuration file and a script to export all objects from the Oracle database. Here a sample of the command usage: ora2pg --project_base /app/migration/ --init_project test_project Creating project test_project. /app/migration/test_project/ schema/ dblinks/ directories/ functions/ grants/ mviews/ packages/ partitions/ procedures/ sequences/ synonyms/ tables/ tablespaces/ triggers/ types/ views/ sources/ functions/ mviews/ packages/ partitions/ procedures/ triggers/ types/ views/ data/ config/ reports/ Generating generic configuration file Creating script export_schema.sh to automate all exports. Creating script import_all.sh to automate all imports. It create a generic config file where you just have to define the Oracle database connection and a shell script called export_schema.sh. The sources/ directory will contains the Oracle code, the schema/ will contains the code ported to PostgreSQL. The reports/ directory will contains the html and json reports with the migration cost assessment. If you want to use your own default config file, use the -c option to give the path to that file. Rename it with .dist suffix if you want ora2pg to apply the generic configuration values otherwise, the configuration file will be copied untouched. Once you have set the connection to the Oracle Database you can execute the script export_schema.sh that will export all object type from your Oracle database and output DDL files into the schema's subdirectories. At end of the export it will give you the command to export data later when the import of the schema will be done and verified. You can choose to load the DDL files generated manually or use the second script import_all.sh to import those file interactively. If this kind of migration is not something current for you it's recommended you to use those scripts. =head2 Oracle database connection There's 5 configuration directives to control the access to the Oracle database. =over 4 =item ORACLE_HOME Used to set ORACLE_HOME environment variable to the Oracle libraries required by the DBD::Oracle Perl module. =item ORACLE_DSN This directive is used to set the data source name in the form standard DBI DSN. For example: dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 or dbi:Oracle:DB_SID On 18c this could be for example: dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 for the second notation the SID should be declared in the well known file $ORACLE_HOME/network/admin/tnsnames.ora or in the path given to the TNS_ADMIN environment variable. For MySQL the DSN will lool like this: dbi:mysql:host=192.168.1.10;database=sakila;port=3306 the 'sid' part is replaced by 'database'. For MS SQL Server it will look like this: dbi:ODBC:driver=msodbcsql18;server=mydb.database.windows.net;database=testdb;TrustServerCertificate=yes =item ORACLE_USER et ORACLE_PWD These two directives are used to define the user and password for the Oracle database connection. Note that if you can it is better to login as Oracle super admin to avoid grants problem during the database scan and be sure that nothing is missing. If you do not supply a credential with ORACLE_PWD and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If ORACLE_USER is not set it will be asked interactively too. To connect to a local ORACLE instance with connections "as sysdba" you have to set ORACLE_USER to "/" and an empty password. To make a connection using an Oracle Secure External Password Store (SEPS), first configure the Oracle Wallet and then set both the ORACLE_USER and ORACLE_PWD directives to the special value of "__SEPS__" (without the quotes but with the double underscore). =item USER_GRANTS Set this directive to 1 if you connect the Oracle database as simple user and do not have enough grants to extract things from the DBA_... tables. It will use tables ALL_... instead. Warning: if you use export type GRANT, you must set this configuration option to 0 or it will not work. =item TRANSACTION This directive may be used if you want to change the default isolation level of the data export transaction. Default is now to set the level to a serializable transaction to ensure data consistency. The allowed values for this directive are: readonly: 'SET TRANSACTION READ ONLY', readwrite: 'SET TRANSACTION READ WRITE', serializable: 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' committed: 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED', Releases before 6.2 used to set the isolation level to READ ONLY transaction but in some case this was breaking data consistency so now default is set to SERIALIZABLE. =item INPUT_FILE This directive did not control the Oracle database connection or unless it purely disables the use of any Oracle database by accepting a file as argument. Set this directive to a file containing PL/SQL Oracle Code like function, procedure or full package body to prevent Ora2Pg from connecting to an Oracle database and just apply his conversion tool to the content of the file. This can be used with the most of export types: TABLE, TRIGGER, PROCEDURE, VIEW, FUNCTION or PACKAGE, etc. =item ORA_INITIAL_COMMAND This directive can be used to send an initial command to Oracle, just after the connection. For example to unlock a policy before reading objects or to set some session parameters. This directive can be used multiple times. =back =head2 Data encryption with Oracle server If your Oracle Client config file already includes the encryption method, then DBD:Oracle uses those settings to encrypt the connection while you extract the data. For example if you have configured the Oracle Client config file (sqlnet.ora or .sqlnet) with the following information: # Configure encryption of connections to Oracle SQLNET.ENCRYPTION_CLIENT = required SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) SQLNET.CRYPTO_SEED = 'should be 10-70 random characters' Any tool that uses the Oracle client to talk to the database will be encrypted if you setup session encryption like above. For example, Perl's DBI uses DBD-Oracle, which uses the Oracle client for actually handling database communication. If the installation of Oracle client used by Perl is setup to request encrypted connections, then your Perl connection to an Oracle database will also be encrypted. Full details at https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 =head2 Testing connection Once you have set the Oracle database DSN you can execute ora2pg to see if it works: ora2pg -t SHOW_VERSION -c config/ora2pg.conf will show the Oracle database server version. Take some time here to test your installation as most problems take place here, the other configuration steps are more technical. =head2 Troubleshooting If the output.sql file has not exported anything other than the Pg transaction header and footer there's two possible reasons. The perl script ora2pg dump an ORA-XXX error, that mean that your DSN or login information are wrong, check the error and your settings and try again. The perl script says nothing and the output file is empty: the user lacks permission to extract something from the database. Try to connect to Oracle as super user or take a look at directive USER_GRANTS above and at next section, especially the SCHEMA directive. =over 4 =item LOGFILE By default all messages are sent to the standard output. If you give a file path to that directive, all output will be appended to this file. =back =head2 Oracle schema to export The Oracle database export can be limited to a specific Schema or Namespace, this can be mandatory following the database connection user. =over 4 =item SCHEMA This directive is used to set the schema name to use during export. For example: SCHEMA APPS will extract objects associated to the APPS schema. When no schema name is provided and EXPORT_SCHEMA is enabled, Ora2Pg will export all objects from all schema of the Oracle instance with their names prefixed with the schema name. =item EXPORT_SCHEMA By default the Oracle schema is not exported into the PostgreSQL database and all objects are created under the default Pg namespace. If you want to also export this schema and create all objects under this namespace, set the EXPORT_SCHEMA directive to 1. This will set the schema search_path at top of export SQL file to the schema name set in the SCHEMA directive with the default pg_catalog schema. If you want to change this path, use the directive PG_SCHEMA. =item CREATE_SCHEMA Enable/disable the CREATE SCHEMA SQL order at starting of the output file. It is enable by default and concern on TABLE export type. =item COMPILE_SCHEMA By default Ora2Pg will only export valid PL/SQL code. You can force Oracle to compile again the invalidated code to get a chance to have it obtain the valid status and then be able to export it. Enable this directive to force Oracle to compile schema before exporting code. When this directive is enabled and SCHEMA is set to a specific schema name, only invalid objects in this schema will be recompiled. If SCHEMA is not set then all schema will be recompiled. To force recompile invalid object in a specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. This will ask to Oracle to validate the PL/SQL that could have been invalidate after a export/import for example. The 'VALID' or 'INVALID' status applies to functions, procedures, packages and user defined types. It also concern disabled triggers. =item EXPORT_INVALID If the above configuration directive is not enough to validate your PL/SQL code enable this configuration directive to allow export of all PL/SQL code even if it is marked as invalid. The 'VALID' or 'INVALID' status applies to functions, procedures, packages, triggers and user defined types. =item PG_SCHEMA Allow you to defined/force the PostgreSQL schema to use. By default if you set EXPORT_SCHEMA to 1 the PostgreSQL search_path will be set to the schema name exported set as value of the SCHEMA directive. The value can be a comma delimited list of schema name but not when using TABLE export type because in this case it will generate the CREATE SCHEMA statement and it doesn't support multiple schema name. For example, if you set PG_SCHEMA to something like "user_schema, public", the search path will be set like this: SET search_path = user_schema, public; forcing the use of an other schema (here user_schema) than the one from Oracle schema set in the SCHEMA directive. You can also set the default search_path for the PostgreSQL user you are using to connect to the destination database by using: ALTER ROLE username SET search_path TO user_schema, public; in this case you don't have to set PG_SCHEMA. =item SYSUSERS Without explicit schema, Ora2Pg will export all objects that not belongs to system schema or role: SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER Following your Oracle installation you may have several other system role defined. To append these users to the schema exclusion list, just set the SYSUSERS configuration directive to a comma-separated list of system user to exclude. For example: SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH will add users INTERNAL and SYSDBA to the schema exclusion list. =item FORCE_OWNER By default the owner of the database objects is the one you're using to connect to PostgreSQL using the psql command. If you use an other user (postgres for example) you can force Ora2Pg to set the object owner to be the one used in the Oracle database by setting the directive to 1, or to a completely different username by setting the directive value to that username. =item FORCE_SECURITY_INVOKER Ora2Pg use the function's security privileges set in Oracle and it is often defined as SECURITY DEFINER. If you want to override those security privileges for all functions and use SECURITY DEFINER instead, enable this directive. =item USE_TABLESPACE When enabled this directive force ora2pg to export all tables, indexes constraint and indexes using the tablespace name defined in Oracle database. This works only with tablespace that are not TEMP, USERS and SYSTEM. =item WITH_OID Activating this directive will force Ora2Pg to add WITH (OIDS) when creating tables or views as tables. Default is same as PostgreSQL, disabled. =item LOOK_FORWARD_FUNCTION List of schema to get functions/procedures meta information that are used in the current schema export. When replacing call to function with OUT parameters, if a function is declared in an other package then the function call rewriting can not be done because Ora2Pg only knows about functions declared in the current schema. By setting a comma separated list of schema as value of this directive, Ora2Pg will look forward in these packages for all functions/procedures/packages declaration before proceeding to current schema export. =item NO_FUNCTION_METADATA Force Ora2Pg to not look for function declaration. Note that this will prevent Ora2Pg to rewrite function replacement call if needed. Do not enable it unless looking forward at function breaks other export. =back =head2 Export type The export action is perform following a single configuration directive 'TYPE', some other add more control on what should be really exported. =over 4 =item TYPE Here are the different values of the TYPE directive, default is TABLE: - TABLE: Extract all tables with indexes, primary keys, unique keys, foreign keys and check constraints. - VIEW: Extract only views. - GRANT: Extract roles converted to Pg groups, users and grants on all objects. - SEQUENCE: Extract all sequence and their last position. - TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). - TRIGGER: Extract triggers defined following actions. - FUNCTION: Extract functions. - PROCEDURE: Extract procedures. - PACKAGE: Extract packages and package bodies. - INSERT: Extract data as INSERT statement. - COPY: Extract data as COPY statement. - PARTITION: Extract range and list Oracle partitions with subpartitions. - TYPE: Extract user defined Oracle type. - FDW: Export Oracle tables as foreign table for Oracle, MySQL and SQL Server FDW. - MVIEW: Export materialized view. - QUERY: Try to automatically convert Oracle SQL queries. - KETTLE: Generate XML ktr template files to be used by Kettle. - DBLINK: Generate oracle foreign data wrapper server to use as dblink. - SYNONYM: Export Oracle's synonyms as views on other schema's objects. - DIRECTORY: Export Oracle's directories as external_file extension objects. - LOAD: Dispatch a list of queries over multiple PostgreSQl connections. - TEST: perform a diff between Oracle and PostgreSQL database. - TEST_COUNT: perform a row count diff between Oracle and PostgreSQL table. - TEST_VIEW: perform a count on both side of number of rows returned by views. - TEST_DATA: perform data validation check on rows at both sides. - SEQUENCE_VALUES: export DDL to set the last values of sequences Only one type of export can be perform at the same time so the TYPE directive must be unique. If you have more than one only the last found in the file will be registered. Some export type can not or should not be load directly into the PostgreSQL database and still require little manual editing. This is the case for GRANT, TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, QUERY and PACKAGE export types especially if you have PLSQL code or Oracle specific SQL in it. For TABLESPACE you must ensure that file path exist on the system and for SYNONYM you may ensure that the object's owners and schemas correspond to the new PostgreSQL database design. Note that you can chained multiple export by giving to the TYPE directive a comma-separated list of export type, but in this case you must not use COPY or INSERT with other export type. Ora2Pg will convert Oracle partition using table inheritance, trigger and functions. See document at Pg site: http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html The TYPE export allow export of user defined Oracle type. If you don't use the --plsql command line parameter it simply dump Oracle user type asis else Ora2Pg will try to convert it to PostgreSQL syntax. The KETTLE export type requires that the Oracle and PostgreSQL DNS are defined. Since Ora2Pg v8.1 there's three new export types: SHOW_VERSION : display Oracle version SHOW_SCHEMA : display the list of schema available in the database. SHOW_TABLE : display the list of tables available. SHOW_COLUMN : display the list of tables columns available and the Ora2PG conversion type from Oracle to PostgreSQL that will be applied. It will also warn you if there's PostgreSQL reserved words in Oracle object names. Here is an example of the SHOW_COLUMN output: [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: 'CURRENT_SCHEMA' is a reserved word in PostgreSQL) CONSTRAINT : NUMBER(22) => bigint (Warning: 'CONSTRAINT' is a reserved word in PostgreSQL) FREEZE : VARCHAR2(25) => varchar(25) (Warning: 'FREEZE' is a reserved word in PostgreSQL) ... [6] TABLE LOCATIONS (23 rows) LOCATION_ID : NUMBER(4) => smallint STREET_ADDRESS : VARCHAR2(40) => varchar(40) POSTAL_CODE : VARCHAR2(12) => varchar(12) CITY : VARCHAR2(30) => varchar(30) STATE_PROVINCE : VARCHAR2(25) => varchar(25) COUNTRY_ID : CHAR(2) => char(2) Those extraction keywords are use to only display the requested information and exit. This allows you to quickly know on what you are going to work. The SHOW_COLUMN allow an other ora2pg command line option: '--allow relname' or '-a relname' to limit the displayed information to the given table. The SHOW_ENCODING export type will display the NLS_LANG and CLIENT_ENCODING values that Ora2Pg will used and the real encoding of the Oracle database with the corresponding client encoding that could be used with PostgreSQL Ora2Pg allow you to export your Oracle, MySQL or MSSQL table definition to be use with the oracle_fdw, mysql_fdw or tds_fdw foreign data wrapper. By using type FDW your tables will be exported as follow: CREATE FOREIGN TABLE oratab ( id integer NOT NULL, text character varying(30), floating double precision NOT NULL ) SERVER oradb OPTIONS (table 'ORATAB'); Now you can use the table like a regular PostgreSQL table. Release 10 adds a new export type destined to evaluate the content of the database to migrate, in terms of objects and cost to end the migration: SHOW_REPORT : show a detailed report of the Oracle database content. Here is a sample of report: http://ora2pg.darold.net/report.html There also a more advanced report with migration cost. See the dedicated chapter about Migration Cost Evaluation. =item ESTIMATE_COST Activate the migration cost evaluation. Must only be used with SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE and QUERY export type. Default is disabled. You may want to use the --estimate_cost command line option instead to activate this functionality. Note that enabling this directive will force PLSQL_PGSQL activation. =item COST_UNIT_VALUE Set the value in minutes of the migration cost evaluation unit. Default is five minutes per unit. See --cost_unit_value to change the unit value at command line. =item DUMP_AS_HTML By default when using SHOW_REPORT the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in HTML format. See http://ora2pg.darold.net/report.html for a sample report. =item DUMP_AS_JSON By default when using SHOW_REPORT the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in JSON format. See http://ora2pg.darold.net/report.html for a sample report. =item DUMP_AS_CSV By default when using SHOW_REPORT the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in CSV format. See http://ora2pg.darold.net/report.html for a sample report. =item DUMP_AS_FILE_PREFIX By default when using SHOW_REPORT the migration report is generated to stout. Enabling this directive in conjunction with DUMP_AS_* directives will force ora2pg to create a report files with the given extensions and formats. This option allows you to combine multiple DUMP_AS_* formats. See http://ora2pg.darold.net/report.html for a sample report. =item HUMAN_DAYS_LIMIT Use this directive to redefined the number of person-days limit where the migration assessment level must switch from B to C. Default is set to 10 person-days. =item JOBS This configuration directive adds multiprocess support to COPY, FUNCTION and PROCEDURE export type, the value is the number of process to use. Default is multiprocess disable. This directive is used to set the number of cores to used to parallelize data import into PostgreSQL. During FUNCTION or PROCEDURE export type each function will be translated to plpgsql using a new process, the performances gain can be very important when you have tons of function to convert. There's no limitation in parallel processing than the number of cores and the PostgreSQL I/O performance capabilities. Doesn't work under Windows Operating System, it is simply disabled. =item ORACLE_COPIES This configuration directive adds multiprocess support to extract data from Oracle. The value is the number of process to use to parallelize the select query. Default is parallel query disable. The parallelism is built on splitting the query following of the number of cores given as value to ORACLE_COPIES as follow: SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC where COLUMN is a technical key like a primary or unique key where split will be based and the current core used by the query (CUR_PROC). You can also force the column name to use using the DEFINED_PK configuration directive. Doesn't work under Windows Operating System, it is simply disabled. =item DEFINED_PK This directive is used to defined the technical key to used to split the query between number of cores set with the ORACLE_COPIES variable. For example: DEFINED_PK EMPLOYEES:employee_id The parallel query that will be used supposing that -J or ORACLE_COPIES is set to 8: SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N where N is the current process forked starting from 0. =item PARALLEL_TABLES This directive is used to defined the number of tables that will be processed in parallel for data extraction. The limit is the number of cores on your machine. Ora2Pg will open one database connection for each parallel table extraction. This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS, so the real number of process that will be used is PARALLEL_TABLES * JOBS. Note that this directive when set upper that 1 will also automatically enable the FILE_PER_TABLE directive if your are exporting to files. This is used to export tables and views in separate files. Use PARALLEL_TABLES to use parallelism with COPY, INSERT and TEST_DATA actions. It is also useful with TEST, TEST_COUNT, and SHOW_TABLE if --count_rows is used for real row count. =item DEFAULT_PARALLELISM_DEGREE You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in each query used to export data from Oracle by setting a value upper than 1 to this directive. A value of 0 or 1 disable the use of parallel hint. Default is disabled. =item FDW_SERVER This directive is used to set the name of the foreign data server that is used in the "CREATE SERVER name FOREIGN DATA WRAPPER ..." command. This name will then be used in the "CREATE FOREIGN TABLE ..." SQL commands and to import data using oracle_fdw. Default is no foreign server defined. This only concerns export type FDW, COPY and INSERT. For export type FDW the default value is orcl. =item FDW_IMPORT_SCHEMA Schema where foreign tables for data migration will be created. If you use several instances of ora2pg for data migration through the foreign data wrapper, you might need to change the name of the schema for each instance. Default: ora2pg_fdw_import =item ORACLE_FDW_PREFETCH The default behaviour of Ora2Pg is to NOT set the "prefetch" option for oracle_fdw when used for COPY and INSERT. This directive allows the prefetch to be set. See oracle_fdw documentation for the current default. =item ORACLE_FDW_COPY_MODE When using Ora2Pg COPY with oracle_fdw it is possible to use two different modes: 1) "local", which uses psql on the host running Ora2Pg for the "TO" binary stream; 2) "server", which uses PostgreSQL server-side COPY for the "TO" binary stream. Both modes use psql for the "FROM STDIN BINARY". However, "local" runs the psql "FROM STDIN BINARY" on host Ora2Pg is run from, whereas "server" runs the psql "FROM STDIN BINARY" on the PostgreSQL server. "local" mode should work on any PostgreSQL-based system, including managed offerings, which are not expected to support use of "server" mode due to permissions. The default is "local" as this is compatible with more configurations. =item ORACLE_FDW_COPY_FORMAT When using Ora2Pg COPY with oracle_fdw it is possible to use either BINARY or CSV data format. BINARY provides better performance, however, requires exact data type matching between the FDW and destination table. CSV provides greater flexibiliity with respect to data type matching: if the FDW and destination data types are functionally-compatible the columns can be copied. The default is "binary". =item DROP_FOREIGN_SCHEMA By default Ora2Pg drops the temporary schema ora2pg_fdw_import used to import the Oracle foreign schema before each new import. If you want to preserve the existing schema because of modifications or the use of a third party server, disable this directive. =item EXTERNAL_TO_FDW This directive, enabled by default, allow to export Oracle's External Tables as file_fdw foreign tables. To not export these tables at all, set the directive to 0. =item INTERNAL_DATE_MAX Internal timestamp retrieves from custom type are extracted in the following format: 01-JAN-77 12.00.00.000000 AM. It is impossible to know the exact century that must be used, so by default any year below 49 will be added to 2000 and others to 1900. You can use this directive to change the default value 49. this is only relevant if you have user defined type with a column timestamp. =item AUDIT_USER Set the comma separated list of username that must be used to filter queries from the DBA_AUDIT_TRAIL table. Default is to not scan this table and to never look for queries. This parameter is used only with SHOW_REPORT and QUERY export type with no input file for queries. Note that queries will be normalized before output unlike when a file is given at input using the -i option or INPUT directive. =item FUNCTION_CHECK Disable this directive if you want to disable check_function_bodies. SET check_function_bodies = false; It disables validation of the function body string during CREATE FUNCTION. Default is to use de postgresql.conf setting that enable it by default. =item ENABLE_BLOB_EXPORT Exporting BLOB takes time, in some circumstances you may want to export all data except the BLOB columns. In this case disable this directive and the BLOB columns will not be included into data export. Take care that the target bytea column do not have a NOT NULL constraint. =item ENABLE_CLOB_EXPORT Same behavior as ENABLE_BLOB_EXPORT but for CLOB. =item DATA_EXPORT_ORDER By default data export order will be done by sorting on table name. If you have huge tables at end of alphabetic order and you are using multiprocess, it can be better to set the sort order on size so that multiple small tables can be processed before the largest tables finish. In this case set this directive to size. Possible values are name and size. Note that export type SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or INSERT export type. If you want to give you custom export order, just give a filename as value that contains the ordered list of table to export. Must be a list of one table per line, in uppercase for Oracle. =back =head2 Limiting objects to export You may want to export only a part of an Oracle database, here are a set of configuration directives that will allow you to control what parts of the database should be exported. =over 4 =item ALLOW This directive allows you to set a list of objects on which the export must be limited, excluding all other objects in the same type of export. The value is a space or comma-separated list of objects name to export. You can include valid regex into the list. For example: ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ will export objects with name EMPLOYEES, COUNTRIES, all objects beginning with 'SALE_' and all objects with a name ending by '_GEOM_SEQ'. The object depends of the export type. Note that regex will not works with 8i database, you must use the % placeholder instead, Ora2Pg will use the LIKE operator. This is the manner to declare global filters that will be used with the current export type. You can also use extended filters that will be applied on specific objects or only on their related export type. For example: ora2pg -p -c ora2pg.conf -t TRIGGER -a 'TABLE[employees]' will limit export of trigger to those defined on table employees. If you want to extract all triggers but not some INSTEAD OF triggers: ora2pg -c ora2pg.conf -t TRIGGER -e 'VIEW[trg_view_.*]' Or a more complex form: ora2pg -p -c ora2pg.conf -t TABLE -a 'TABLE[EMPLOYEES]' \ -e 'INDEX[emp_.*];CKEY[emp_salary_min]' This command will export the definition of the employee table but will exclude all index beginning with 'emp_' and the CHECK constraint called 'emp_salary_min'. When exporting partition you can exclude some partition tables by using ora2pg -p -c ora2pg.conf -t PARTITION -e 'PARTITION[PART_199.* PART_198.*]' This will exclude partitioned tables for year 1980 to 1999 from the export but not the main partition table. The trigger will also be adapted to exclude those table. With GRANT export you can use this extended form to exclude some users from the export or limit the export to some others: ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' or ora2pg -p -c ora2pg.conf -t GRANT -a 'GRANT[USER1 USER2]' will limit export grants to users USER1 and USER2. But if you don't want to export grants on some functions for these users, for example: ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' -e 'FUNCTION[adm_.*];PROCEDURE[adm_.*]' Advanced filters may need some learning. Oracle doesn't allow the use of lookahead expression so you may want to exclude some object that match the ALLOW regexp you have defined. For example if you want to export all table starting with E but not those starting with EXP it is not possible to do that in a single expression. This is why you can start a regular expression with the ! character to exclude object matching the regexp given just after. Our previous example can be written as follow: ALLOW E.* !EXP.* it will be translated into: REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') in the object search expression. =item EXCLUDE This directive is the opposite of the previous, it allow you to define a space or comma-separated list of object name to exclude from the export. You can include valid regex into the list. For example: EXCLUDE EMPLOYEES TMP_.* COUNTRIES will exclude object with name EMPLOYEES, COUNTRIES and all tables beginning with 'tmp_'. For example, you can ban from export some unwanted function with this directive: EXCLUDE write_to_.* send_mail_.* this example will exclude all functions, procedures or functions in a package with the name beginning with those regex. Note that regex will not work with 8i database, you must use the % placeholder instead, Ora2Pg will use the NOT LIKE operator. See above (directive 'ALLOW') for the extended syntax. =item NO_EXCLUDED_TABLE By default Ora2Pg exclude from export some Oracle "garbage" tables that should never be part of an export. This behavior generates a lot of REGEXP_LIKE expressions which are slowing down the export when looking at tables. To disable this behavior enable this directive, you will have to exclude or clean up later by yourself the unwanted tables. The regexp used to exclude the table are defined in the array @EXCLUDED_TABLES in lib/Ora2Pg.pm. Note this is behavior is independant to the EXCLUDE configuration directive. =item VIEW_AS_TABLE Set which view to export as table. By default none. Value must be a list of view name or regexp separated by space or comma. If the object name is a view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. See chapter "Exporting views as PostgreSQL table" for more details. =item MVIEW_AS_TABLE Set which materialized view to export as table. By default none. Value must be a list of materialized view name or regexp separated by space or comma. If the object name is a materialized view and the export type is TABLE, the view will be exported as a create table statement. If export type is COPY or INSERT, the corresponding data will be exported. =item NO_VIEW_ORDERING By default Ora2Pg try to order views to avoid error at import time with nested views. With a huge number of views this can take a very long time, you can bypass this ordering by enabling this directive. =item GRANT_OBJECT When exporting GRANTs you can specify a comma separated list of objects for which privilege will be exported. Default is export for all objects. Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY. Only one object type is allowed at a time. For example set it to TABLE if you just want to export privilege on tables. You can use the -g option to overwrite it. When used this directive prevent the export of users unless it is set to USER. In this case only users definitions are exported. =item WHERE This directive allows you to specify a WHERE clause filter when dumping the contents of tables. Value is constructs as follows: TABLE_NAME[WHERE_CLAUSE], or if you have only one where clause for each table just put the where clause as the value. Both are possible too. Here are some examples: # Global where clause applying to all tables included in the export WHERE 1=1 # Apply the where clause only on table TABLE_NAME WHERE TABLE_NAME[ID1='001'] # Applies two different clause on tables TABLE_NAME and OTHER_TABLE # and a generic where clause on DATE_CREATE to all other tables WHERE TABLE_NAME[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' OTHER_TABLE[NAME='test'] Any where clause not included into a table name bracket clause will be applied to all exported table including the tables defined in the where clause. These WHERE clauses are very useful if you want to archive some data or at the opposite only export some recent data. To be able to quickly test data import it is useful to limit data export to the first thousand tuples of each table. For Oracle define the following clause: WHERE ROWNUM < 1000 and for MySQL, use the following: WHERE 1=1 LIMIT 1,1000 This can also be restricted to some tables data export. Command line option -W or --where will override this directive for the global part and per table if the table names is the same. =item TOP_MAX This directive is used to limit the number of item shown in the top N lists like the top list of tables per number of rows and the top list of largest tables in megabytes. By default it is set to 10 items. =item LOG_ON_ERROR Enable this directive if you want to continue direct data import on error. When Ora2Pg received an error in the COPY or INSERT statement from PostgreSQL it will log the statement to a file called TABLENAME_error.log in the output directory and continue to next bulk of data. Like this you can try to fix the statement and manually reload the error log file. Default is disabled: abort import on error. =item REPLACE_QUERY Sometime you may want to extract data from an Oracle table but you need a custom query for that. Not just a "SELECT * FROM table" like Ora2Pg do but a more complex query. This directive allows you to overwrite the query used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. If you have multiple table to extract by replacing the Ora2Pg query, you can define multiple REPLACE_QUERY lines. REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] =back =head2 Control of Full Text Search export Several directives can be used to control the way Ora2Pg will export the Oracle's Text search indexes. By default CONTEXT indexes will be exported to PostgreSQL FTS indexes but CTXCAT indexes will be exported as indexes using the pg_trgm extension. =over 4 =item CONTEXT_AS_TRGM Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes and CTXCAT indexes using pg_trgm. Most of the time using pg_trgm is enough, this is why this directive stand for. You need to create the pg_trgm extension into the destination database before importing the objects: CREATE EXTENSION pg_trgm; =item FTS_INDEX_ONLY By default Ora2Pg creates a function-based index to translate Oracle Text indexes. CREATE INDEX ON t_document USING gin(to_tsvector('pg_catalog.french', title)); You will have to rewrite the CONTAIN() clause using to_tsvector(), example: SELECT id,title FROM t_document WHERE to_tsvector(title)) @@ to_tsquery('search_word'); To force Ora2Pg to create an extra tsvector column with a dedicated triggers for FTS indexes, disable this directive. In this case, Ora2Pg will add the column as follow: ALTER TABLE t_document ADD COLUMN tsv_title tsvector; Then update the column to compute FTS vectors if data have been loaded before UPDATE t_document SET tsv_title = to_tsvector('pg_catalog.french', coalesce(title,'')); To automatically update the column when a modification in the title column appears, Ora2Pg adds the following trigger: CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ BEGIN IF TG_OP = 'INSERT' OR new.title != old.title THEN new.tsv_title := to_tsvector('pg_catalog.french', coalesce(new.title,'')); END IF; return new; END $$ LANGUAGE plpgsql; CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE ON t_document FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); When the Oracle text index is defined over multiple column, Ora2Pg will use setweight() to set a weight in the order of the column declaration. =item FTS_CONFIG Use this directive to force text search configuration to use. When it is not set, Ora2Pg will autodetect the stemmer used by Oracle for each index and pg_catalog.english if the information is not found. =item USE_UNACCENT If you want to perform your text search in an accent insensitive way, enable this directive. Ora2Pg will create an helper function over unaccent() and creates the pg_trgm indexes using this function. With FTS Ora2Pg will redefine your text search configuration, for example: CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; then set the FTS_CONFIG ora2pg.conf directive to fr instead of pg_catalog.english. When enabled, Ora2pg will create the wrapper function: CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$ SELECT public.unaccent('public.unaccent', $1); $$ LANGUAGE sql IMMUTABLE COST 1; the indexes are exported as follow: CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document USING gin (unaccent_immutable(title) gin_trgm_ops); In your queries you will need to use the same function in the search to be able to use the function-based index. Example: SELECT * FROM t_document WHERE unaccent_immutable(title) LIKE '%donnees%'; =item USE_LOWER_UNACCENT Same as above but call lower() in the unaccent_immutable() function: CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS $$ SELECT lower(public.unaccent('public.unaccent', $1)); $$ LANGUAGE sql IMMUTABLE; =back =head2 Modifying object structure One of the great usage of Ora2Pg is its flexibility to replicate Oracle database into PostgreSQL database with a different structure or schema. There's three configuration directives that allow you to map those differences. =over 4 =item REORDERING_COLUMNS Enable this directive to reordering columns and minimized the footprint on disc, so that more rows fit on a data page, which is the most important factor for speed. Default is disabled, that mean the same order than in Oracle tables definition, that's should be enough for most usage. This directive is only used with TABLE export. =item MODIFY_STRUCT This directive allows you to limit the columns to extract for a given table. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... for example: MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) This will only extract columns 'id' and 'dossier' from table T_TEST1 and columns 'id' and 'fichier' from the T_TEST2 table. This directive can only be used with TABLE, COPY or INSERT export. With TABLE export create table DDL will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. =item EXCLUDE_COLUMNS Instead of redefining the table structure with MODIFY_STRUCT you may want to exclude some columns from the table export. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: EXCLUDE_COLUMNS NOM_TABLE(nomcol1,nomcol2,...) ... for example: EXCLUDE_COLUMNS T_TEST1(id,dossier) T_TEST2(id,fichier) This will exclude from the export columns 'id' and 'dossier' from table T_TEST1 and columns 'id' and 'fichier' from the T_TEST2 table. This directive can only be used with TABLE, COPY or INSERT export. With TABLE export create table DDL will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. =item REPLACE_TABLES This directive allows you to remap a list of Oracle table name to a PostgreSQL table name during export. The value is a list of space-separated values with the following structure: REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 Oracle tables ORIG_TBNAME1 and ORIG_TBNAME2 will be respectively renamed into DEST_TBNAME1 and DEST_TBNAME2 =item REPLACE_COLS Like table name, the name of the column can be remapped to a different name using the following syntax: REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) For example: REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) will rename Oracle columns 'dico' and 'dossier' from table T_TEST into new name 'dictionary' and 'folder'. =item REPLACE_AS_BOOLEAN If you want to change the type of some Oracle columns into PostgreSQL boolean during the export you can define here a list of tables and column separated by space as follow. REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 The values set in the boolean columns list will be replaced with the 't' and 'f' following the default replacement values and those additionally set in directive BOOLEAN_VALUES. Note that if you have modified the table name with REPLACE_TABLES and/or the column's name, you need to use the name of the original table and/or column. REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 You can also give a type and a precision to automatically convert all fields of that type as a boolean. For example: REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 will also replace any field of type number(1) or char(1) as a boolean in all exported tables. =item BOOLEAN_VALUES Use this to add additional definition of the possible boolean values used in Oracle fields. You must set a space-separated list of TRUE:FALSE values. By default here are the values recognized by Ora2Pg: BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled Any values defined here will be added to the default list. =item REPLACE_ZERO_DATE When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL. This could be a problem if your column is defined with NOT NULL constraint. If you can not remove the constraint, use this directive to set an arbitral date that will be used instead. You can also use -INFINITY if you don't want to use a fake date. =item INDEXES_SUFFIX Add the given value as suffix to indexes names. Useful if you have indexes with same name as tables. For example: INDEXES_SUFFIX _idx will add _idx at ed of all index name. Not so common but can help. =item INDEXES_RENAMING Enable this directive to rename all indexes using tablename_columns_names. Could be very useful for database that have multiple time the same index name or that use the same name than a table, which is not allowed by PostgreSQL Disabled by default. =item USE_INDEX_OPCLASS Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B-tree indexes on the corresponding types. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (LIKE or POSIX regular expressions) when the database does not use the standard "C" locale. If you enable, with value 1, this will force Ora2Pg to export all indexes defined on varchar2() and char() columns using those operators. If you set it to a value greater than 1 it will only change indexes on columns where the character limit is greater or equal than this value. For example, set it to 128 to create these kind of indexes on columns of type varchar2(N) where N >= 128. =item RENAME_PARTITION Enable this directive if you want that your partition tables will be renamed. Disabled by default. If you have multiple partitioned table, when exported to PostgreSQL some partitions could have the same name but different parent tables. This is not allowed, table name must be unique, in this case enable this directive. A partition will be renamed following the rule: "tablename"_part"pos" where "pos" is the partition number. For subpartition this is: "tablename"_part"pos"_subpart"pos" If this is partition/subpartition default: "tablename"_part_default "tablename"_part"pos"_subpart_default =item DISABLE_PARTITION If you don't want to reproduce the partitioning like in Oracle and want to export all partitioned Oracle data into the main single table in PostgreSQL enable this directive. Ora2Pg will export all data into the main table name. Default is to use partitioning, Ora2Pg will export data from each partition and import them into the PostgreSQL dedicated partition table. =item PARTITION_BY_REFERENCE How to export partition by reference. Possible values are none, duplicate or the number of hash partition to create. Default is none to not export the partitions by reference. Value none mean no translation and export of partition by reference like before. Value 'duplicate' will duplicate the referenced column in the partitioned table and apply the same partitioning from the referenced table to the partitioned table. If the value is a number, the table will be partitioned with the HASH method using the value as the modulo. For example if you set it to 4 it will create 4 HASH partitions. =item DISABLE_UNLOGGED By default Ora2Pg export Oracle tables with the NOLOGGING attribute as UNLOGGED tables. You may want to fully disable this feature because you will lose all data from unlogged tables in case of a PostgreSQL crash. Set it to 1 to export all tables as normal tables. =item DOUBLE_MAX_VARCHAR Increase varchar max character constraints to support PostgreSQL two bytes character encoding when the source database applies the length constraint on characters not bytes. Default disabled. =back =head2 Oracle Spatial to PostGis Ora2Pg fully export Spatial object from Oracle database. There's some configuration directives that could be used to control the export. =over 4 =item AUTODETECT_SPATIAL_TYPE By default Ora2Pg is looking at indexes to see the spatial constraint type and dimensions defined under Oracle. Those constraints are passed as at index creation using for example: CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS('sdo_indx_dims=2, layer_gtype=point'); If those Oracle constraints parameters are not set, the default is to export those columns as generic type GEOMETRY to be able to receive any spatial type. The AUTODETECT_SPATIAL_TYPE directive allows to force Ora2Pg to autodetect the real spatial type and dimension used in a spatial column otherwise a non- constrained "geometry" type is used. Enabling this feature will force Ora2Pg to scan a sample of 50000 column to look at the GTYPE used. You can increase or reduce the sample size by setting the value of AUTODETECT_SPATIAL_TYPE to the desired number of line to scan. The directive is enabled by default. For example, in the case of a column named shape and defined with Oracle type SDO_GEOMETRY, with AUTODETECT_SPATIAL_TYPE disabled it will be converted as: shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) and if the directive is enabled and the column just contains a single geometry type that use a single dimension: shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) with a two or three dimensional polygon. =item CONVERT_SRID This directive allows you to control the automatically conversion of Oracle SRID to standard EPSG. If enabled, Ora2Pg will use the Oracle function sdo_cs.map_oracle_srid_to_epsg() to convert all SRID. Enabled by default. If the SDO_SRID returned by Oracle is NULL, it will be replaced by the default value 8307 converted to its EPSG value: 4326 (see DEFAULT_SRID). If the value is upper than 1, all SRID will be forced to this value, in this case DEFAULT_SRID will not be used when Oracle returns a null value and the value will be forced to CONVERT_SRID. Note that it is also possible to set the EPSG value on Oracle side when sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to force the value: system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; =item DEFAULT_SRID Use this directive to override the default EPSG SRID to used: 4326. Can be overwritten by CONVERT_SRID, see above. =item GEOMETRY_EXTRACT_TYPE This directive can take three values: WKT (default), WKB and INTERNAL. When it is set to WKT, Ora2Pg will use SDO_UTIL.TO_WKTGEOMETRY() to extract the geometry data. When it is set to WKB, Ora2Pg will use the binary output using SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type are calls at Oracle side, they are slow and you can easily reach Out Of Memory when you have lot of rows. Also WKB is not able to export 3D geometry and some geometries like CURVEPOLYGON. In this case you may use the INTERNAL extraction type. It will use a Pure Perl library to convert the SDO_GEOMETRY data into a WKT representation, the translation is done on Ora2Pg side. This is a work in progress, please validate your exported data geometries before use. Default spatial object extraction type is INTERNAL. =item POSTGIS_SCHEMA Use this directive to add a specific schema to the search path to look for PostGis functions. =item ST_SRID_FUNCTION Oracle function to use to extract the srid from ST_Geometry meta information. Default: ST_SRID, for example it should be set to sde.st_srid for ArcSDE. =item ST_DIMENSION_FUNCTION Oracle function to use to extract the dimension from ST_Geometry meta information. Default: ST_DIMENSION, for example it should be set to sde.st_dimention for ArcSDE. =item ST_GEOMETRYTYPE_FUNCTION Oracle function to use to extract the geometry type from a ST_Geometry column Default: ST_GEOMETRYTYPE, for example it should be set to sde.st_geometrytype for ArcSDE. =item ST_ASBINARY_FUNCTION Oracle function to used to convert an ST_Geometry value into WKB format. Default: ST_ASBINARY, for example it should be set to sde.st_asbinary for ArcSDE. =item ST_ASTEXT_FUNCTION Oracle function to used to convert an ST_Geometry value into WKT format. Default: ST_ASTEXT, for example it should be set to sde.st_astext for ArcSDE. =back =head2 PostgreSQL Import By default conversion to PostgreSQL format is written to file 'output.sql'. The command: psql mydb < output.sql will import content of file output.sql into PostgreSQL mydb database. =over 4 =item DATA_LIMIT When you are performing INSERT/COPY export Ora2Pg proceed by chunks of DATA_LIMIT tuples for speed improvement. Tuples are stored in memory before being written to disk, so if you want speed and have enough system resources you can grow this limit to an upper value for example: 100000 or 1000000. Before release 7.0 a value of 0 mean no limit so that all tuples are stored in memory before being flushed to disk. In 7.x branch this has been remove and chunk will be set to the default: 10000 =item BLOB_LIMIT When Ora2Pg detect a table with some BLOB it will automatically reduce the value of this directive by dividing it by 10 until his value is below 1000. You can control this value by setting BLOB_LIMIT. Exporting BLOB use lot of resources, setting it to a too high value can produce OOM. =item CLOB_AS_BLOB Apply same behavior on CLOB than BLOB with BLOB_LIMIT settings. This is especially useful if you have large CLOB data. Default: enabled =item OUTPUT The Ora2Pg output filename can be changed with this directive. Default value is output.sql. if you set the file name with extension .gz or .bz2 the output will be automatically compressed. This require that the Compress::Zlib Perl module is installed if the filename extension is .gz and that the bzip2 system command is installed for the .bz2 extension. =item OUTPUT_DIR Since release 7.0, you can define a base directory where the file will be written. The directory must exists. =item BZIP2 This directive allows you to specify the full path to the bzip2 program if it can not be found in the PATH environment variable. =item FILE_PER_CONSTRAINT Allow object constraints to be saved in a separate file during schema export. The file will be named CONSTRAINTS_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can use .gz xor .bz2 extension to enable compression. Default is to save all data in the OUTPUT file. This directive is usable only with TABLE export type. The constraints can be imported quickly into PostgreSQL using the LOAD export type to parallelize their creation over multiple (-j or JOBS) connections. =item FILE_PER_INDEX Allow indexes to be saved in a separate file during schema export. The file will be named INDEXES_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can use .gz xor .bz2 file extension to enable compression. Default is to save all data in the OUTPUT file. This directive is usable only with TABLE AND TABLESPACE export type. With the TABLESPACE export, it is used to write "ALTER INDEX ... TABLESPACE ..." into a separate file named TBSP_INDEXES_OUTPUT that can be loaded at end of the migration after the indexes creation to move the indexes. The indexes can be imported quickly into PostgreSQL using the LOAD export type to parallelize their creation over multiple (-j or JOBS) connections. =item FILE_PER_FKEYS Allow foreign key declaration to be saved in a separate file during schema export. By default foreign keys are exported into the main output file or in the CONSTRAINT_output.sql file. When enabled foreign keys will be exported into a file named FKEYS_output.sql =item FILE_PER_TABLE Allow data export to be saved in one file per table/view. The files will be named as tablename_OUTPUT, where OUTPUT is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the OUTPUT directive to enable compression. Default 0 will save all data in one file, set it to 1 to enable this feature. This is usable only during INSERT or COPY export type. =item FILE_PER_FUNCTION Allow functions, procedures and triggers to be saved in one file per object. The files will be named as objectname_OUTPUT. Where OUTPUT is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the OUTPUT directive to enable compression. Default 0 will save all in one single file, set it to 1 to enable this feature. This is usable only during the corresponding export type, the package body export has a special behavior. When export type is PACKAGE and you've enabled this directive, Ora2Pg will create a directory per package, named with the lower case name of the package, and will create one file per function/procedure into that directory. If the configuration directive is not enabled, it will create one file per package as packagename_OUTPUT, where OUTPUT is the value of the corresponding directive. =item TRUNCATE_TABLE If this directive is set to 1, a TRUNCATE TABLE instruction will be add before loading data. This is usable only during INSERT or COPY export type. When activated, the instruction will be added only if there's no global DELETE clause or not one specific to the current table (see below). =item DELETE Support for include a DELETE FROM ... WHERE clause filter before importing data and perform a delete of some lines instead of truncating tables. Value is construct as follow: TABLE_NAME[DELETE_WHERE_CLAUSE], or if you have only one where clause for all tables just put the delete clause as single value. Both are possible too. Here are some examples: DELETE 1=1 # Apply to all tables and delete all tuples DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] The last applies two different delete where clause on tables TABLE_TEST and TABLE_INFO and a generic delete where clause on DATE_CREATE to all other tables. If TRUNCATE_TABLE is enabled it will be applied to all tables not covered by the DELETE definition. These DELETE clauses might be useful with regular "updates". =item STOP_ON_ERROR Set this parameter to 0 to not include the call to \set ON_ERROR_STOP ON in all SQL scripts generated by Ora2Pg. By default this order is always present so that the script will immediately abort when an error is encountered. =item COPY_FREEZE Enable this directive to use COPY FREEZE instead of a simple COPY to export data with rows already frozen. This is intended as a performance option for initial data loading. Rows will be frozen only if the table being loaded has been created or truncated in the current sub-transaction. This will only work with export to file and when -J or ORACLE_COPIES is not set or default to 1. It can be used with direct import into PostgreSQL under the same condition but -j or JOBS must also be unset or default to 1. =item CREATE_OR_REPLACE By default Ora2Pg uses CREATE OR REPLACE in functions and views DDL, if you need not to override existing functions or views disable this configuration directive, DDL will not include OR REPLACE. =item DROP_IF_EXISTS To add a DROP IF EXISTS before creating the object, enable this directive. Can be useful in an iterative work. Default is disabled. =item EXPORT_GTT PostgreSQL do not supports Global Temporary Table natively but you can use the pgtt extension to emulate this behavior. Enable this directive to export global temporary table. =item PGTT_NOSUPERUSER By default the pgtt extension is loaded using the superuser privilege. Enabled it if you run the SQL scripts generated using a non superuser user. It will use: LOAD '$libdir/plugins/pgtt'; instead of default: LOAD 'pgtt'; =item NO_HEADER Enabling this directive will prevent Ora2Pg to print his header into output files. Only the translated code will be written. =item PSQL_RELATIVE_PATH By default Ora2Pg use \i psql command to execute generated SQL files if you want to use a relative path following the script execution file enabling this option will use \ir. See psql help for more information. =item DATA_VALIDATION_ROWS Number of rows that must be retrieved on both side for data validation. Default it to compare the 10000 first rows. A value of 0 mean compare all rows. =item DATA_VALIDATION_ORDERING Order of rows between both sides are different once the data have been modified. In this case data must be ordered using a primary key or a unique index, that mean that a table without such object can not be compared. If the validation is done just after the data migration without any data modification the validation can be done on all tables without any ordering. =item DATA_VALIDATION_ERROR Stop validating data from a table after a certain amount of row mistmatch. Default is to stop after 10 rows validation errors. =item TRANSFORM_VALUE Use this directive to precise which transformation should be applied to a column when exporting data. Value must be a semicolon separated list of TABLE[COLUMN_NAME, ] For example to replace string 'Oracle' by 'PostgreSQL' in a varchar2 column use the following. TRANSFORM_VALUE ERROR_LOG_SAMPLE[DBMS_TYPE:regexp_replace("DBMS_TYPE",'Oracle','PostgreSQL')] or to replace all Oracle char(0) in a string by a space character: TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), ' ')] The expression will be applied in the SQL statemeent used to extract data from the source database. =back When using Ora2Pg export type INSERT or COPY to dump data to file and that FILE_PER_TABLE is enabled, you will be warned that Ora2Pg will not export data again if the file already exists. This is to prevent downloading twice table with huge amount of data. To force the download of data from these tables you have to remove the existing output file first. If you want to import data on the fly to the PostgreSQL database you have three configuration directives to set the PostgreSQL database connection. This is only possible with COPY or INSERT export type as for database schema there's no real interest to do that. =over 4 =item PG_DSN Use this directive to set the PostgreSQL data source namespace using DBD::Pg Perl module as follow: dbi:Pg:dbname=pgdb;host=localhost;port=5432 will connect to database 'pgdb' on localhost at tcp port 5432. Note that this directive is only used for data export, other export need to be imported manually through the use og psql or any other PostgreSQL client. To use SSL encrypted connection you must add sslmode=require to the connection string like follow: dbi:Pg:dbname=pgdb;host=localhost;port=5432;sslmode=require =item PG_USER and PG_PWD These two directives are used to set the login user and password. If you do not supply a credential with PG_PWD and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If PG_USER is not set it will be asked interactively too. =item SYNCHRONOUS_COMMIT Specifies whether transaction commit will wait for WAL records to be written to disk before the command returns a "success" indication to the client. This is the equivalent to set synchronous_commit directive of postgresql.conf file. This is only used when you load data directly to PostgreSQL, the default is off to disable synchronous commit to gain speed at writing data. Some modified version of PostgreSQL, like greenplum, do not have this setting, so in this set this directive to 1, ora2pg will not try to change the setting. =item PG_INITIAL_COMMAND This directive can be used to send an initial command to PostgreSQL, just after the connection. For example to set some session parameters. This directive can be used multiple times. =item INSERT_ON_CONFLICT When enabled this instruct Ora2Pg to add an ON CONFLICT DO NOTHING clause to all INSERT statements generated for this type of data export. =back =head2 Column type control =over 4 =item PG_NUMERIC_TYPE If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type NUMBER(p,s) is approximatively converted to real and float PostgreSQL data type. If you have monetary fields or don't want rounding issues with the extra decimals you should preserve the same numeric(p,s) PostgreSQL data type. Do that only if you need exactness because using numeric(p,s) is slower than using real or double. =item PG_INTEGER_TYPE If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type NUMBER(p) or NUMBER are converted to smallint, integer or bigint PostgreSQL data type following the value of the precision. If NUMBER without precision are set to DEFAULT_NUMERIC (see below). =item DEFAULT_NUMERIC NUMBER without precision are converted by default to bigint only if PG_INTEGER_TYPE is true. You can overwrite this value to any PG type, like integer or float. =item DATA_TYPE If you're experiencing any problem in data type schema conversion with this directive you can take full control of the correspondence between Oracle and PostgreSQL types to redefine data type translation used in Ora2pg. The syntax is a comma-separated list of "Oracle datatype:Postgresql datatype". Here are the default list used: DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,NVARCHAR:varchar,NCHAR:char,DATE:timestamp(0),LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW(16):uuid,RAW(32):uuid,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:integer,INTEGER:integer,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone The directive and the list definition must be a single line. Note that when a RAW(16) and RAW(32) columns is found or that the RAW column has "SYS_GUID()" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the "uuid-ossp" extension. If you want to replace a type with a precision and scale you need to escape the coma with a backslash. For example, if you want to replace all NUMBER(*,0) into bigint instead of numeric(38) add the following: DATA_TYPE NUMBER(*\,0):bigint You don't have to recopy all default type conversion but just the one you want to rewrite. There's a special case with BFILE when they are converted to type TEXT, they will just contains the full path to the external file. If you set the destination type to BYTEA, the default, Ora2Pg will export the content of the BFILE as bytea. The third case is when you set the destination type to EFILE, in this case, Ora2Pg will export it as an EFILE record: (DIRECTORY, FILENAME). Use the DIRECTORY export type to export the existing directories as well as privileges on those directories. There's no SQL function available to retrieve the path to the BFILE. Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); l_path VARCHAR2(4000); BEGIN dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); SELECT directory_path INTO l_path FROM all_directories WHERE directory_name = l_dir; l_dir := rtrim(l_path,'/'); RETURN l_dir || '/' || l_fname; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is TEXT. The function is dropped at the end of the export. This concern both, COPY and INSERT export type. There's no SQL function available to retrieve BFILE as an EFILE record, then Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); BEGIN dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); RETURN '(' || l_dir || ',' || l_fnamei || ')'; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is EFILE. The function is dropped at the end of the export. This concern both, COPY and INSERT export type. To set the destination type, use the DATA_TYPE configuration directive: DATA_TYPE BFILE:EFILE for example. The EFILE type is a user defined type created by the PostgreSQL extension external_file that can be found here: https://github.com/darold/external_file This is a port of the BFILE Oracle type to PostgreSQL. There's no SQL function available to retrieve the content of a BFILE. Ora2Pg have to create one using the DBMS_LOB package. CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN BLOB AS filecontent BLOB := NULL; src_file BFILE := NULL; l_step PLS_INTEGER := 12000; l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); offset NUMBER := 1; BEGIN IF p_bfile IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); src_file := BFILENAME( l_dir, l_fname ); IF src_file IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); DBMS_LOB.CREATETEMPORARY(filecontent, true); DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); DBMS_LOB.FILECLOSE(src_file); RETURN filecontent; END; This function is only created if Ora2Pg found a table with a BFILE column and that the destination type is bytea (the default). The function is dropped at the end of the export. This concern both, COPY and INSERT export type. About the ROWID and UROWID, they are converted into OID by "logical" default but this will through an error at data import. There is no equivalent data type so you might want to use the DATA_TYPE directive to change the corresponding type in PostgreSQL. You should consider replacing this data type by a bigserial (autoincremented sequence), text or uuid data type. =item MODIFY_TYPE Sometimes you need to force the destination type, for example a column exported as timestamp by Ora2Pg can be forced into type date. Value is a comma-separated list of TABLE:COLUMN:TYPE structure. If you need to use comma or space inside type definition you will have to backslash them. MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) Type of table1.col3 will be replaced by a varchar and table1.col4 by a decimal with precision and scale. If the column's type is a user defined type Ora2Pg will autodetect the composite type and will export its data using ROW(). Some Oracle user defined types are just array of a native type, in this case you may want to transform this column in simple array of a PostgreSQL native type. To do so, just redefine the destination type as wanted and Ora2Pg will also transform the data as an array. For example, with the following definition in Oracle: CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); CREATE TABLE club (Name VARCHAR2(10), Address VARCHAR2(20), City VARCHAR2(20), Phone VARCHAR2(8), Members mem_type ); custom type "mem_type" is just a string array and can be translated into the following in PostgreSQL: CREATE TABLE club ( name varchar(10), address varchar(20), city varchar(20), phone varchar(8), members text[] ) ; To do so, just use the directive as follow: MODIFY_TYPE CLUB:MEMBERS:text[] Ora2Pg will take care to transform all data of this column in the correct format. Only arrays of characters and numerics types are supported. =item TO_NUMBER_CONVERSION By default Oracle call to function TO_NUMBER will be translated as a cast into numeric. For example, TO_NUMBER('10.1234') is converted into PostgreSQL call to_number('10.1234')::numeric. If you want you can cast the call to integer or bigint by changing the value of the configuration directive. If you need better control of the format, just set it as value, for example: TO_NUMBER_CONVERSION 99999999999999999999.9999999999 will convert the code above as: TO_NUMBER('10.1234', '99999999999999999999.9999999999') Any value of the directive that it is not numeric, integer or bigint will be taken as a mask format. If set to none, no conversion will be done. =item VARCHAR_TO_TEXT By default varchar2 without size constraint are tranlated into text. If you want to keep the varchar name, disable this directive. =item FORCE_IDENTITY_BIGINT Usually identity column must be bigint to correspond to an auto increment sequence so Ora2Pg always force it to be a bigint. If, for any reason you want Ora2Pg to respect the DATA_TYPE you have set for identity column then disable this directive. =item TO_CHAR_NOTIMEZONE If you want Ora2Pg to remove any timezone information into the format part of the TO_CHAR() function, enable this directive. Disabled by default. =back =head2 Taking export under control The following other configuration directives interact directly with the export process and give you fine granularity in database export control. =over 4 =item SKIP For TABLE export you may not want to export all schema constraints, the SKIP configuration directive allows you to specify a space-separated list of constraints that should not be exported. Possible values are: - fkeys: turn off foreign key constraints - pkeys: turn off primary keys - ukeys: turn off unique column constraints - indexes: turn off all other index types - checks: turn off check constraints For example: SKIP indexes,checks will removed indexes and check constraints from export. =item PKEY_IN_CREATE Enable this directive if you want to add primary key definition inside the create table statement. If disabled (the default) primary key definition will be added with an alter table statement. Enable it if you are exporting to GreenPlum PostgreSQL database. =item KEEP_PKEY_NAMES By default names of the primary and unique key in the source Oracle database are ignored and key names are autogenerated in the target PostgreSQL database with the PostgreSQL internal default naming rules. If you want to preserve Oracle primary and unique key names set this option to 1. =item FKEY_ADD_UPDATE This directive allows you to add an ON UPDATE CASCADE option to a foreign key when a ON DELETE CASCADE is defined or always. Oracle do not support this feature, you have to use trigger to operate the ON UPDATE CASCADE. As PostgreSQL has this feature, you can choose how to add the foreign key option. There are three values to this directive: never, the default that mean that foreign keys will be declared exactly like in Oracle. The second value is delete, that mean that the ON UPDATE CASCADE option will be added only if the ON DELETE CASCADE is already defined on the foreign Keys. The last value, always, will force all foreign keys to be defined using the update option. =item FKEY_DEFERRABLE When exporting tables, Ora2Pg normally exports constraints as they are, if they are non-deferrable they are exported as non-deferrable. However, non-deferrable constraints will probably cause problems when attempting to import data to Pg. The FKEY_DEFERRABLE option set to 1 will cause all foreign key constraints to be exported as deferrable. =item DEFER_FKEY In addition to exporting data when the DEFER_FKEY option set to 1, it will add a command to defer all foreign key constraints during data export and the import will be done in a single transaction. This will work only if foreign keys have been exported as deferrable and you are not using direct import to PostgreSQL (PG_DSN is not defined). Constraints will then be checked at the end of the transaction. This directive can also be enabled if you want to force all foreign keys to be created as deferrable and initially deferred during schema export (TABLE export type). =item DROP_FKEY If deferring foreign keys is not possible due to the amount of data in a single transaction, you've not exported foreign keys as deferrable or you are using direct import to PostgreSQL, you can use the DROP_FKEY directive. It will drop all foreign keys before all data import and recreate them at the end of the import. =item DROP_INDEXES This directive allows you to gain lot of speed improvement during data import by removing all indexes that are not an automatic index (indexes of primary keys) and recreate them at the end of data import. Of course it is far better to not import indexes and constraints before having imported all data. =item DISABLE_TRIGGERS This directive is used to disable triggers on all tables in COPY or INSERT export modes. Available values are USER (disable user-defined triggers only) and ALL (includes RI system triggers). Default is 0: do not add SQL statements to disable trigger before data import. If you want to disable triggers during data migration, set the value to USER if your are connected as non superuser and ALL if you are connected as PostgreSQL superuser. A value of 1 is equal to USER. =item DISABLE_SEQUENCE If set to 1 it disables alter of sequences on all tables during COPY or INSERT export mode. This is used to prevent the update of sequence during data migration. Default is 0, alter sequences. =item NOESCAPE By default all data that are not of type date or time are escaped. If you experience any problem with that you can set it to 1 to disable character escaping during data export. This directive is only used during a COPY export. See STANDARD_CONFORMING_STRINGS for enabling/disabling escape with INSERT statements. =item STANDARD_CONFORMING_STRINGS This controls whether ordinary string literals ('...') treat backslashes literally, as specified in SQL standard. This was the default before Ora2Pg v8.5 so that all strings was escaped first, now this is currently on, causing Ora2Pg to use the escape string syntax (E'...') if this parameter is not set to 0. This is the exact behavior of the same option in PostgreSQL. This directive is only used during data export to build INSERT statements. See NOESCAPE for enabling/disabling escape in COPY statements. =item TRIM_TYPE If you want to convert CHAR(n) from Oracle into varchar(n) or text on PostgreSQL using directive DATA_TYPE, you might want to do some trimming on the data. By default Ora2Pg will auto-detect this conversion and remove any whitespace at both leading and trailing position. If you just want to remove the leadings character set the value to LEADING. If you just want to remove the trailing character, set the value to TRAILING. Default value is BOTH. =item TRIM_CHAR The default trimming character is space, use this directive if you need to change the character that will be removed. For example, set it to - if you have leading - in the char(n) field. To use space as trimming charger, comment this directive, this is the default value. =item PRESERVE_CASE If you want to preserve the case of Oracle object name set this directive to 1. By default Ora2Pg will convert all Oracle object names to lower case. I do not recommend to enable this unless you will always have to double-quote object names on all your SQL scripts. =item ORA_RESERVED_WORDS Allow escaping of column name using Oracle reserved words. Value is a list of comma-separated reserved word. Default: audit,comment,references. =item USE_RESERVED_WORDS Enable this directive if you have table or column names that are a reserved word for PostgreSQL. Ora2Pg will double quote the name of the object. =item GEN_USER_PWD Set this directive to 1 to replace default password by a random password for all extracted user during a GRANT export. =item PG_SUPPORTS_MVIEW Since PostgreSQL 9.3, materialized view are supported with the SQL syntax 'CREATE MATERIALIZED VIEW'. To force Ora2Pg to use the native PostgreSQL support you must enable this configuration - enable by default. If you want to use the old style with table and a set of function, you should disable it. =item PG_SUPPORTS_IFEXISTS PostgreSQL version below 9.x do not support IF EXISTS in DDL statements. Disabling the directive with value 0 will prevent Ora2Pg to add those keywords in all generated statements. Default value is 1, enabled. =item PG_VERSION Set the PostgreSQL major version number of the target database. Ex: 9.6 or 13 Default is current major version at time of a new release. This replace the old and deprecadted PG_SUPPORTS_* configuration directives described bellow. =item PG_SUPPORTS_ROLE (Deprecated) This option is deprecated since Ora2Pg release v7.3. By default Oracle roles are translated into PostgreSQL groups. If you have PostgreSQL 8.1 or more consider the use of ROLES and set this directive to 1 to export roles. =item PG_SUPPORTS_INOUT (Deprecated) This option is deprecated since Ora2Pg release v7.3. If set to 0, all IN, OUT or INOUT parameters will not be used into the generated PostgreSQL function declarations (disable it for PostgreSQL database version lower than 8.1), This is now enable by default. =item PG_SUPPORTS_DEFAULT This directive enable or disable the use of default parameter value in function export. Until PostgreSQL 8.4 such a default value was not supported, this feature is now enable by default. =item PG_SUPPORTS_WHEN (Deprecated) Add support to WHEN clause on triggers as PostgreSQL v9.0 now support it. This directive is enabled by default, set it to 0 disable this feature. =item PG_SUPPORTS_INSTEADOF (Deprecated) Add support to INSTEAD OF usage on triggers (used with PG >= 9.1), if this directive is disabled the INSTEAD OF triggers will be rewritten as Pg rules. =item PG_SUPPORTS_CHECKOPTION When enabled, export views with CHECK OPTION. Disable it if you have PostgreSQL version prior to 9.4. Default: 1, enabled. =item PG_SUPPORTS_IFEXISTS If disabled, do not export object with IF EXISTS statements. Enabled by default. =item PG_SUPPORTS_PARTITION PostgreSQL version prior to 10.0 do not have native partitioning. Enable this directive if you want to use declarative partitioning. Enable by default. =item PG_SUPPORTS_SUBSTR Some versions of PostgreSQL like Redshift doesn't support substr() and it need to be replaced by a call to substring(). In this case, disable it. =item PG_SUPPORTS_NAMED_OPERATOR Disable this directive if you are using PG < 9.5, PL/SQL operator used in named parameter => will be replaced by PostgreSQL proprietary operator := Enable by default. =item PG_SUPPORTS_IDENTITY Enable this directive if you have PostgreSQL >= 10 to use IDENTITY columns instead of serial or bigserial data type. If PG_SUPPORTS_IDENTITY is disabled and there is IDENTITY column in the Oracle table, they are exported as serial or bigserial columns. When it is enabled they are exported as IDENTITY columns like: CREATE TABLE identity_test_tab ( id bigint GENERATED ALWAYS AS IDENTITY, description varchar(30) ) ; If there is non default sequence options set in Oracle, they will be appended after the IDENTITY keyword. Additionally in both cases, Ora2Pg will create a file AUTOINCREMENT_output.sql with a embedded function to update the associated sequences with the restart value set to "SELECT max(colname)+1 FROM tablename". Of course this file must be imported after data import otherwise sequence will be kept to start value. Enabled by default. =item PG_SUPPORTS_PROCEDURE PostgreSQL v11 adds support of PROCEDURE, enable it if you use such version. =item BITMAP_AS_GIN Use btree_gin extension to create bitmap like index with pg >= 9.4 You will need to create the extension by yourself: create extension btree_gin; Default is to create GIN index, when disabled, a btree index will be created =item PG_BACKGROUND Use pg_background extension to create an autonomous transaction instead of using a dblink wrapper. With pg >= 9.5 only. Default is to use dblink. See https://github.com/vibhorkum/pg_background about this extension. =item DBLINK_CONN By default if you have an autonomous transaction translated using dblink extension instead of pg_background the connection is defined using the values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override the connection string use this directive as follow to set the connection in the autonomous transaction wrapper function. For example: DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass =item LONGREADLEN Use this directive to set the database handle's 'LongReadLen' attribute to a value that will be the larger than the expected size of the LOBs. The default is 1MB witch may not be enough to extract BLOBs or CLOBs. If the size of the LOB exceeds the 'LongReadLen' DBD::Oracle will return a 'ORA-24345: A Truncation' error. Default: 1023*1024 bytes. Take a look at this page to learn more: http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs Important note: If you increase the value of this directive take care that DATA_LIMIT will probably needs to be reduced. Even if you only have a 1MB blob, trying to read 10000 of them (the default DATA_LIMIT) all at once will require 10GB of memory. You may extract data from those table separately and set a DATA_LIMIT to 500 or lower, otherwise you may experience some out of memory. =item LONGTRUNKOK If you want to bypass the 'ORA-24345: A Truncation' error, set this directive to 1, it will truncate the data extracted to the LongReadLen value. Disable by default so that you will be warned if your LongReadLen value is not high enough. =item USE_LOB_LOCATOR Disable this if you want to load full content of BLOB and CLOB and not use LOB locators. In this case you will have to set LONGREADLEN to the right value. Note that this will not improve speed of BLOB export as most of the time is always consumed by the bytea escaping and in this case export is done line by line and not by chunk of DATA_LIMIT rows. For more information on how it works, see http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators Default is enabled, it use LOB locators. =item LOB_CHUNK_SIZE Oracle recommends reading from and writing to a LOB in batches using a multiple of the LOB chunk size. This chunk size defaults to 8k (8192). Recent tests shown that the best performances can be reach with higher value like 512K or 4Mb. A quick benchmark with 30120 rows with different size of BLOB (200x5Mb, 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, LONGREADLEN=170Mb and a total table size of 20GB gives: no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE set to 4Mb. Depending of the size of most BLOB you may want to adjust the value here. For example if you have a majority of small lobs bellow 8K, using 8192 is better to not waste space. Default value for LOB_CHUNK_SIZE is 512000. =item XML_PRETTY Force the use getStringVal() instead of getClobVal() for XML data export. Default is 1, enabled for backward compatibility. Set it to 0 to use extract method a la CLOB. Note that XML value extracted with getStringVal() must not exceed VARCHAR2 size limit (4000) otherwise it will return an error. =item ENABLE_MICROSECOND Set it to O if you want to disable export of millisecond from Oracle timestamp columns. By default milliseconds are exported with the use of following format: 'YYYY-MM-DD HH24:MI:SS.FF' Disabling will force the use of the following Oracle format: to_char(..., 'YYYY-MM-DD HH24:MI:SS') By default milliseconds are exported. =item DISABLE_COMMENT Set this to 1 if you don't want to export comment associated to tables and columns definition. Default is enabled. =back =head2 Control MySQL export behavior =over 4 =item MYSQL_PIPES_AS_CONCAT Enable this if double pipe and double ampersand (|| and &&) should not be taken as equivalent to OR and AND. It depend of the variable @sql_mode, Use it only if Ora2Pg fail on auto detecting this behavior. =item MYSQL_INTERNAL_EXTRACT_FORMAT Enable this directive if you want EXTRACT() replacement to use the internal format returned as an integer, for example DD HH24:MM:SS will be replaced with format; DDHH24MMSS::bigint, this depend of your apps usage. =back =head2 Control SQL Server export behavior =over 4 =item DROP_ROWVERSION PostgreSQL has no equivalent to rowversion datatype and feature, if you want to remove these useless columns, enable this directive. Columns of datatype 'rowversion' or 'timestamp' will not be exported. =item CASE_INSENSITIVE_SEARCH Emulate the same behavior of MSSQL with case insensitive search. If the value is citext it will use the citext data type instead of char/varchar/text in tables DDL (Ora2Pg will add a CHECK constraint for columns with a precision). Instead of citext you can also set a collation name that will be used in the columns definitions. To disable case insensitive search set it to: none. =item SELECT_TOP Append a TOP N clause to the SELECT command used to extract the data from SQL Server. This is the equivalent to a WHERE ROWNUM < 1000 clause for Oracle. =back =head2 Special options to handle character encoding =over 4 =item NLS_LANG and NLS_NCHAR By default Ora2Pg will set NLS_LANG to AMERICAN_AMERICA.AL32UTF8 and NLS_NCHAR to AL32UTF8. It is not recommended to change those settings but in some case it could be useful. Using your own settings with those configuration directive will change the client encoding at Oracle side by setting the environment variables $ENV{NLS_LANG} and $ENV{NLS_NCHAR}. =item BINMODE By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through a call to the Perl pragma: use open ':utf8'; You can override this encoding by using the BINMODE directive, for example you can set it to :locale to use your locale or iso-8859-7, it will respectively use use open ':locale'; use open ':encoding(iso-8859-7)'; If you have change the NLS_LANG in non UTF8 encoding, you might want to set this directive. See http://perldoc.perl.org/5.14.2/open.html for more information. Most of the time, leave this directive commented. =item CLIENT_ENCODING By default PostgreSQL client encoding is automatically set to UTF8 to avoid encoding issue. If you have changed the value of NLS_LANG you might have to change the encoding of the PostgreSQL client. You can take a look at the PostgreSQL supported character sets here: http://www.postgresql.org/docs/9.0/static/multibyte.html =item FORCE_PLSQL_ENCODING To force utf8 encoding of the PL/SQL code exported, enable this directive. Could be helpful in some rare condition. =back =head2 PLSQL to PLPGSQL conversion Automatic code conversion from Oracle PLSQL to PostgreSQL PLPGSQL is a work in progress in Ora2Pg and surely you will always have manual work. The Perl code used for automatic conversion is all stored in a specific Perl Module named Ora2Pg/PLSQL.pm feel free to modify/add you own code and send me patches. The main work in on function, procedure, package and package body headers and parameters rewrite. =over 4 =item PLSQL_PGSQL Enable/disable PLSQL to PLPGSQL conversion. Enabled by default. =item NULL_EQUAL_EMPTY Ora2Pg can replace all conditions with a test on NULL by a call to the coalesce() function to mimic the Oracle behavior where empty string are considered equal to NULL. (field1 IS NULL) is replaced by (coalesce(field1::text, '') = '') (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> '') You might want this replacement to be sure that your application will have the same behavior but if you have control on you application a better way is to change it to transform empty string into NULL because PostgreSQL makes the difference. =item EMPTY_LOB_NULL Force empty_clob() and empty_blob() to be exported as NULL instead as empty string for the first one and '\x' for the second. If NULL is allowed in your column this might improve data export speed if you have lot of empty lob. Default is to preserve the exact data from Oracle. =item PACKAGE_AS_SCHEMA If you don't want to export package as schema but as simple functions you might also want to replace all call to package_name.function_name. If you disable the PACKAGE_AS_SCHEMA directive then Ora2Pg will replace all call to package_name.function_name() by package_name_function_name(). Default is to use a schema to emulate package. The replacement will be done in all kind of DDL or code that is parsed by the PLSQL to PLPGSQL converter. PLSQL_PGSQL must be enabled or -p used in command line. =item REWRITE_OUTER_JOIN Enable this directive if the rewrite of Oracle native syntax (+) of OUTER JOIN is broken. This will force Ora2Pg to not rewrite such code, default is to try to rewrite simple form of right outer join for the moment. =item UUID_FUNCTION By default Ora2Pg will convert call to SYS_GUID() Oracle function with a call to uuid_generate_v4 from uuid-ossp extension. You can redefined it to use the gen_random_uuid function from pgcrypto extension by changing the function name. Default to uuid_generate_v4. Note that when a RAW(16) and RAW(32) columns is found or that the RAW column has "SYS_GUID()" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the "uuid-ossp" extension. =item FUNCTION_STABLE By default Oracle functions are marked as STABLE as they can not modify data unless when used in PL/SQL with variable assignment or as conditional expression. You can force Ora2Pg to create these function as VOLATILE by disabling this configuration directive. =item COMMENT_COMMIT_ROLLBACK By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to force the user to review the logic of the function. Once it is fixed in Oracle source code or you want to comment this calls enable the following directive. =item COMMENT_SAVEPOINT It is common to see SAVEPOINT call inside PL/SQL procedure together with a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is enabled you may want to also comment SAVEPOINT calls, in this case enable it. =item STRING_CONSTANT_REGEXP Ora2Pg replace all string constant during the pl/sql to plpgsql translation, string constant are all text include between single quote. If you have some string placeholder used in dynamic call to queries you can set a list of regexp to be temporary replaced to not break the parser. For example: STRING_CONSTANT_REGEXP The list of regexp must use the semi colon as separator. =item ALTERNATIVE_QUOTING_REGEXP To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals set the regexp with the text capture to use to extract the text part. For example with a variable declared as c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; the regexp to use must be: ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' ora2pg will use the $$ delimiter, with the example the result will be: c_sample varchar(100) := $$This doesn't work.$$; The value of this configuration directive can be a list of regexp separated by a semi colon. The capture part (between parenthesis) is mandatory in each regexp if you want to restore the string constant. =item USE_ORAFCE If you want to use functions defined in the Orafce library and prevent Ora2Pg to translate call to these functions, enable this directive. The Orafce library can be found here: https://github.com/orafce/orafce By default Ora2pg rewrite add_month(), add_year(), date_trunc() and to_char() functions, but you may prefer to use the orafce version of these function that do not need any code transformation. =item AUTONOMOUS_TRANSACTION Enable translation of autonomous transactions into a wrapper function using dblink or pg_background extension. If you don't want to use this translation and just want the function to be exported as a normal one without the pragma call, disable this directive. =back =head2 Materialized view Materialized views are exported as snapshot "Snapshot Materialized Views" as PostgreSQL only supports full refresh. If you want to import the materialized views in PostgreSQL prior to 9.3 you have to set configuration directive PG_SUPPORTS_MVIEW to 0. In this case Ora2Pg will export all materialized views as explain in this document: http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. When exporting materialized view Ora2Pg will first add the SQL code to create the "materialized_views" table: CREATE TABLE materialized_views ( mview_name text NOT NULL PRIMARY KEY, view_name text NOT NULL, iname text, last_refresh TIMESTAMP WITH TIME ZONE ); all materialized views will have an entry in this table. It then adds the plpgsql code to create tree functions: create_materialized_view(text, text, text) used to create a materialized view drop_materialized_view(text) used to delete a materialized view refresh_full_materialized_view(text) used to refresh a view then it adds the SQL code to create the view and the materialized view: CREATE VIEW mviewname_mview AS SELECT ... FROM ...; SELECT create_materialized_view('mviewname','mviewname_mview', change with the name of the column to used for the index); The first argument is the name of the materialized view, the second the name of the view on which the materialized view is based and the third is the column name on which the index should be build (aka most of the time the primary key). This column is not automatically deduced so you need to replace its name. As said above Ora2Pg only supports snapshot materialized views so the table will be entirely refreshed by issuing first a truncate of the table and then by load again all data from the view: refresh_full_materialized_view('mviewname'); To drop the materialized view you just have to call the drop_materialized_view() function with the name of the materialized view as parameter. =head2 Other configuration directives =over 4 =item DEBUG Set it to 1 will enable verbose output. =item IMPORT You can define common Ora2Pg configuration directives into a single file that can be imported into other configuration files with the IMPORT configuration directive as follow: IMPORT commonfile.conf will import all configuration directives defined into commonfile.conf into the current configuration file. =back =head2 Exporting views as PostgreSQL tables You can export any Oracle view as a PostgreSQL table simply by setting TYPE configuration option to TABLE to have the corresponding create table statement. Or use type COPY or INSERT to export the corresponding data. To allow that you have to specify your views in the VIEW_AS_TABLE configuration option. Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) into a PG create table form, then it will extract the data (if TYPE=COPY or INSERT) following the view schema. For example, with the following view: CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS SELECT category_id, COUNT(*) as product_count, MIN(list_price) as low_price, MAX(list_price) as high_price FROM product_information GROUP BY category_id; Setting VIEW_AS_TABLE to product_prices and using export type TABLE, will force Ora2Pg to detect columns returned types and to generate a create table statement: CREATE TABLE product_prices ( category_id bigint, product_count integer, low_price numeric, high_price numeric ); Data will be loaded following the COPY or INSERT export type and the view declaration. You can use the ALLOW and EXCLUDE directive in addition to filter other objects to export. =head2 Export as Kettle transformation XML files The KETTLE export type is useful if you want to use Penthalo Data Integrator (Kettle) to import data to PostgreSQL. With this type of export Ora2Pg will generate one XML Kettle transformation files (.ktr) per table and add a line to manually execute the transformation in the output.sql file. For example: ora2pg -c ora2pg.conf -t KETTLE -j 12 -a MYTABLE -o load_mydata.sh will generate one file called 'HR.MYTABLE.ktr' and add a line to the output file (load_mydata.sh): #!/bin/sh KETTLE_TEMPLATE_PATH='.' JAVAMAXMEM=4096 ./pan.sh -file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr -level Detailed The -j 12 option will create a template with 12 processes to insert data into PostgreSQL. It is also possible to specify the number of parallel queries used to extract data from the Oracle with the -J command line option as follow: ora2pg -c ora2pg.conf -t KETTLE -J 4 -j 12 -a EMPLOYEES -o load_mydata.sh This is only possible if there is a unique key defined on a numeric column or that you have defined the technical key to used to split the query between cores in the DEFINED_PKEY configuration directive. For example: DEFINED_PK EMPLOYEES:employee_id will force the number of Oracle connection copies to 4 and defined the SQL query as follow in the Kettle XML transformation file: SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} The KETTLE export type requires that the Oracle and PostgreSQL DSN are defined. You can also activate the TRUNCATE_TABLE directive to force a truncation of the table before data import. The KETTLE export type is an original work of Marc Cousin. =head2 Migration cost assessment Estimating the cost of a migration process from Oracle to PostgreSQL is not easy. To obtain a good assessment of this migration cost, Ora2Pg will inspect all database objects, all functions and stored procedures to detect if there's still some objects and PL/SQL code that can not be automatically converted by Ora2Pg. Ora2Pg has a content analysis mode that inspect the Oracle database to generate a text report on what the Oracle database contains and what can not be exported. To activate the "analysis and report" mode, you have to use the export de type SHOW_REPORT like in the following command: ora2pg -t SHOW_REPORT Here is a sample report obtained with this command: -------------------------------------- Ora2Pg: Oracle Database Content Report -------------------------------------- Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 Schema HR Size 880.00 MB -------------------------------------- Object Number Invalid Comments -------------------------------------- CLUSTER 2 0 Clusters are not supported and will not be exported. FUNCTION 40 0 Total size of function code: 81992. INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL. 1 bitmap index(es). 230 b-tree index(es). 1 reversed b-tree index(es) Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed. PACKAGE BODY 2 1 Total size of package code: 20700. PROCEDURE 7 0 Total size of procedure code: 19198. SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files. 2 binary columns. 4 unknown types. TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. TRIGGER 30 0 Total size of trigger code: 21677. TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are converted as table, type inheritance is not supported. TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. Once the database can be analysed, Ora2Pg, by his ability to convert SQL and PL/SQL code from Oracle syntax to PostgreSQL, can go further by estimating the code difficulties and estimate the time necessary to operate a full database migration. To estimate the migration cost in person-days, Ora2Pg allow you to use a configuration directive called ESTIMATE_COST that you can also enabled at command line: --estimate_cost This feature can only be used with the SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE and QUERY export type. ora2pg -t SHOW_REPORT --estimate_cost The generated report is same as above but with a new 'Estimated cost' column as follow: -------------------------------------- Ora2Pg: Oracle Database Content Report -------------------------------------- Version Oracle Database 10g Express Edition Release 10.2.0.1.0 Schema HR Size 890.00 MB -------------------------------------- Object Number Invalid Estimated cost Comments -------------------------------------- DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions using oracle_fdw. FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL. 11 b-tree index(es). Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. JOB 0 0 0 Job are not exported. You may set external cron job with them. MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed. PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround is to use views or set the PostgreSQL search_path in your session to access object outside the current schema. user1.emp_details_view_v is an alias to hr.emp_details_view. user1.emp_table is an alias to hr.employees@other_server. user1.offices is an alias to hr.locations. TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files. 2 binary columns. 4 unknown types. TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are converted as table, type inheritance is not supported. TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. -------------------------------------- Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). The last line shows the total estimated migration cost in person-days following the number of migration units estimated for each object. This migration unit represent around five minutes for a PostgreSQL expert. If this is your first migration you can get it higher with the configuration directive COST_UNIT_VALUE or the --cost_unit_value command line option: ora2pg -t SHOW_REPORT --estimate_cost --cost_unit_value 10 Ora2Pg is also able to give you a migration difficulty level assessment, here a sample: Migration level: B-5 Migration levels: A - Migration that might be run automatically B - Migration with code rewrite and a person-days cost up to 5 days C - Migration with code rewrite and a person-days cost above 5 days Technical levels: 1 = trivial: no stored functions and no triggers 2 = easy: no stored functions but with triggers, no manual rewriting 3 = simple: stored functions and/or triggers, no manual rewriting 4 = manual: no stored functions but with triggers or views with code rewriting 5 = difficult: stored functions and/or triggers with code rewriting This assessment consist in a letter A or B to specify if the migration needs manual rewriting or not. And a number from 1 up to 5 to give you a technical difficulty level. You have an additional option --human_days_limit to specify the number of person-days limit where the migration level should be set to C to indicate that it need a huge amount of work and a full project management with migration support. Default is 10 person-days. You can use the configuration directive HUMAN_DAYS_LIMIT to change this default value permanently. This feature has been developed to help you or your boss to decide which database to migrate first and the team that must be mobilized to operate the migration. =head2 Global Oracle and MySQL migration assessment Ora2Pg come with a script ora2pg_scanner that can be used when you have a huge number of instances and schema to scan for migration assessment. Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] -b | --binpath DIR: full path to directory where the ora2pg binary stays. Might be useful only on Windows OS. -c | --config FILE: set custom configuration file to use otherwise ora2pg will use the default: /etc/ora2pg/ora2pg.conf. -l | --list FILE : CSV file containing a list of databases to scan with all required information. The first line of the file can contain the following header that describes the format that must be used: "type","schema/database","dsn","user","password" -o | --outdir DIR : (optional) by default all reports will be dumped to a directory named 'output', it will be created automatically. If you want to change the name of this directory, set the name at second argument. -t | --test : just try all connections by retrieving the required schema or database name. Useful to validate your CSV list file. -u | --unit MIN : redefine globally the migration cost unit value in minutes. Default is taken from the ora2pg.conf (default 5 minutes). Here is a full example of a CSV databases list file: "type","schema/database","dsn","user","password" "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","HR","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","system","manager" The CSV field separator must be a comma. Note that if you want to scan all schemas from an Oracle instance you just have to leave the schema field empty, Ora2Pg will automatically detect all available schemas and generate a report for each one. Of course you need to use a connection user with enough privileges to be able to scan all schemas. For example: "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","usrname","passwd" will generate a report for all schema in the XE instance. Note that in this case the SCHEMA directive in ora2pg.conf must not be set. It will generate a CSV file with the assessment result, one line per schema or database and a detailed HTML report for each database scanned. Hint: Use the -t | --test option before to test all your connections in your CSV file. For Windows users you must use the -b command line option to set the directory where ora2pg_scanner stays otherwise the ora2pg command calls will fail. In the migration assessment details about functions Ora2Pg always include per default 2 migration units for TEST and 1 unit for SIZE per 1000 characters in the code. This mean that by default it will add 15 minutes in the migration assessment per function. Obviously if you have unitary tests or very simple functions this will not represent the real migration time. =head2 Migration assessment method Migration unit scores given to each type of Oracle database object are defined in the Perl library lib/Ora2Pg/PLSQL.pm in the %OBJECT_SCORE variable definition. The number of PL/SQL lines associated to a migration unit is also defined in this file in the $SIZE_SCORE variable value. The number of migration units associated to each PL/SQL code difficulties can be found in the same Perl library lib/Ora2Pg/PLSQL.pm in the hash %UNCOVERED_SCORE initialization. This assessment method is a work in progress so I'm expecting feedbacks on migration experiences to polish the scores/units attributed in those variables. =head2 Improving indexes and constraints creation speed Using the LOAD export type and a file containing SQL orders to perform, it is possible to dispatch those orders over multiple PostgreSQL connections. To be able to use this feature, the PG_DSN, PG_USER and PG_PWD must be set. Then: ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 will dispatch indexes creation over 4 simultaneous PostgreSQL connections. This will considerably accelerate this part of the migration process with huge data size. =head2 Exporting LONG RAW If you still have columns defined as LONG RAW, Ora2Pg will not be able to export these kind of data. The OCI library fail to export them and always return the same first record. To be able to export the data you need to transform the field as BLOB by creating a temporary table before migrating data. For example, the Oracle table: SQL> DESC TEST_LONGRAW Name NULL ? Type -------------------- -------- ---------------------------- ID NUMBER C1 LONG RAW need to be "translated" into a table using BLOB as follow: CREATE TABLE test_blob (id NUMBER, c1 BLOB); And then copy the data with the following INSERT query: INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; Then you just have to exclude the original table from the export (see EXCLUDE directive) and to renamed the new temporary table on the fly using the REPLACE_TABLES configuration directive. =head2 Global variables Oracle allow the use of global variables defined in packages. Ora2Pg will export these variables for PostgreSQL as user defined custom variables available in a session. Oracle variables assignment are exported as call to: PERFORM set_config('pkgname.varname', value, false); Use of these variables in the code is replaced by: current_setting('pkgname.varname')::global_variables_type; where global_variables_type is the type of the variable extracted from the package definition. If the variable is a constant or have a default value assigned at declaration, Ora2Pg will create a file global_variables.conf with the definition to include in the postgresql.conf file so that their values will already be set at database connection. Note that the value can always modified by the user so you can not have exactly a constant. =head2 Hints Converting your queries with Oracle style outer join (+) syntax to ANSI standard SQL at the Oracle side can save you lot of time for the migration. You can use TOAD Query Builder can re-write these using the proper ANSI syntax, see: http://www.toadworld.com/products/toad-for-oracle/f/10/t/9518.aspx There's also an alternative with SQL Developer Data Modeler, see http://www.thatjeffsmith.com/archive/2012/01/sql-developer-data-modeler-quick-tip-use-oracle-join-syntax-or-ansi/ Toad is also able to rewrite the native Oracle DECODE() syntax into ANSI standard SQL CASE statement. You can find some slide about this in a presentation given at PgConf.RU: http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf =head2 Test the migration The type of action called TEST allow you to check that all objects from Oracle database have been created under PostgreSQL. Of course PG_DSN must be set to be able to check PostgreSQL side. Note that this feature respect the schema name limitation if EXPORT_SCHEMA and SCHEMA or PG_SCHEMA are defined. If only EXPORT_SCHEMA is set all schemes from Oracle and PostgreSQL are scanned. You can filter to a single schema using SCHEMA and/or PG_SCHEMA but you can not filter on a list of schema. To test a list of schema you will have to repeat the calls to Ora2Pg by specifying a single schema each time. For example command: ora2pg -t TEST -c config/ora2pg.conf > migration_diff.txt Will create a file containing the report of all object and row count on both side, Oracle and PostgreSQL, with an error section giving you the detail of the differences for each kind of object. Here is a sample result: [TEST INDEXES COUNT] ORACLEDB:DEPARTMENTS:2 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:6 POSTGRES:employees:6 [ERRORS INDEXES COUNT] Table departments doesn't have the same number of indexes in Oracle (2) and in PostgreSQL (1). [TEST UNIQUE CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS UNIQUE CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of unique constraints. [TEST PRIMARY KEYS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS PRIMARY KEYS COUNT] OK, Oracle and PostgreSQL have the same number of primary keys. [TEST CHECK CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS CHECK CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of check constraints. [TEST NOT NULL CONSTRAINTS COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS NOT NULL CONSTRAINTS COUNT] OK, Oracle and PostgreSQL have the same number of not null constraints. [TEST COLUMN DEFAULT VALUE COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS COLUMN DEFAULT VALUE COUNT] OK, Oracle and PostgreSQL have the same number of column default value. [TEST IDENTITY COLUMN COUNT] ORACLEDB:DEPARTMENTS:1 POSTGRES:departments:1 ORACLEDB:EMPLOYEES:0 POSTGRES:employees:0 [ERRORS IDENTITY COLUMN COUNT] OK, Oracle and PostgreSQL have the same number of identity column. [TEST FOREIGN KEYS COUNT] ORACLEDB:DEPARTMENTS:0 POSTGRES:departments:0 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS FOREIGN KEYS COUNT] OK, Oracle and PostgreSQL have the same number of foreign keys. [TEST TABLE COUNT] ORACLEDB:TABLE:2 POSTGRES:TABLE:2 [ERRORS TABLE COUNT] OK, Oracle and PostgreSQL have the same number of TABLE. [TEST TABLE TRIGGERS COUNT] ORACLEDB:DEPARTMENTS:0 POSTGRES:departments:0 ORACLEDB:EMPLOYEES:1 POSTGRES:employees:1 [ERRORS TABLE TRIGGERS COUNT] OK, Oracle and PostgreSQL have the same number of table triggers. [TEST TRIGGER COUNT] ORACLEDB:TRIGGER:2 POSTGRES:TRIGGER:2 [ERRORS TRIGGER COUNT] OK, Oracle and PostgreSQL have the same number of TRIGGER. [TEST VIEW COUNT] ORACLEDB:VIEW:1 POSTGRES:VIEW:1 [ERRORS VIEW COUNT] OK, Oracle and PostgreSQL have the same number of VIEW. [TEST MVIEW COUNT] ORACLEDB:MVIEW:0 POSTGRES:MVIEW:0 [ERRORS MVIEW COUNT] OK, Oracle and PostgreSQL have the same number of MVIEW. [TEST SEQUENCE COUNT] ORACLEDB:SEQUENCE:1 POSTGRES:SEQUENCE:0 [ERRORS SEQUENCE COUNT] SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). [TEST TYPE COUNT] ORACLEDB:TYPE:1 POSTGRES:TYPE:0 [ERRORS TYPE COUNT] TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). [TEST FDW COUNT] ORACLEDB:FDW:0 POSTGRES:FDW:0 [ERRORS FDW COUNT] OK, Oracle and PostgreSQL have the same number of FDW. [TEST FUNCTION COUNT] ORACLEDB:FUNCTION:3 POSTGRES:FUNCTION:3 [ERRORS FUNCTION COUNT] OK, Oracle and PostgreSQL have the same number of functions. [TEST SEQUENCE VALUES] ORACLEDB:EMPLOYEES_NUM_SEQ:1285 POSTGRES:employees_num_seq:1285 [ERRORS SEQUENCE VALUES COUNT] OK, Oracle and PostgreSQL have the same values for sequences [TEST ROWS COUNT] ORACLEDB:DEPARTMENTS:27 POSTGRES:departments:27 ORACLEDB:EMPLOYEES:854 POSTGRES:employees:854 [ERRORS ROWS COUNT] OK, Oracle and PostgreSQL have the same number of rows. =head2 Data validation Data validation consists in comparing data retrieved from a foreign table pointing to the source Oracle table and a local PostgreSQL table resulting from the data export. To run data validation you can use a direct connection like any other Ora2Pg action but you can also use the oracle_fdw, mysql_fdw ior tds_fdw extension provided that FDW_SERVER and PG_DSN configuration directives are set. By default Ora2Pg will extract the 10000 first rows from both side, you can change this value using directive DATA_VALIDATION_ROWS. When it is set to zero all rows of the tables will be compared. Data validation requires that the table has a primary key or unique index and that the key columns is not a LOB. Rows will be sorted using this unique key. Due to differences in sort behavior between Oracle and PostgreSQL, if the collation of unique key columns in PostgreSQL is not 'C', the sort order can be different compared to Oracle. In this case the data validation will fail. Data validation must be done before any data is modified. Ora2Pg will stop comparing two tables after DATA_VALIDATION_ROWS is reached or that 10 errors has been encountered, result is dumped in a file named "data_validation.log" written in the current directory by default. The number of error before stopping the diff between rows can be controlled using the configuration directive DATA_VALIDATION_ERROR. All rows in errors are printed to the output file for your analyze. It is possible to parallelize data validation by using -P option or the corresponding configuration directive PARALLEL_TABLES in ora2pg.conf. =head2 Use of System Change Number (SCN) Ora2Pg is able to export data as of a specific SCN. You can set it at command line using the -S or --scn option. You can give a specific SCN or if you want to use the current SCN at first connection time set the value to 'current'. In this last case the connection user has the "SELECT ANY DICTIONARY" or the "SELECT_CATALOG_ROLE" role, the current SCN is looked at the v$database view. Example of use: ora2pg -c ora2pg.conf -t COPY --scn 16605281 This adds the following clause to the query used to retrieve data for example: AS OF SCN 16605281 You can also use th --scn option to use the Oracle flashback capabality by specifying a timestamp expression instead of a SCN. For example: ora2pg -c ora2pg.conf -t COPY --scn "TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS')" This will add the following clause to the query used to retrieve data: AS OF TIMESTAMP TO_TIMESTAMP('2021-12-01 00:00:00', 'YYYY-MM-DD HH:MI:SS') or for example to only retrive yesterday's data: ora2pg -c ora2pg.conf -t COPY --scn "SYSDATE - 1" =head2 Change Data Capture (CDC) Ora2Pg do not have such feature which allow to import data and to only apply changes after the first import. But you can use the --cdc_ready option to export data with registration of the SCN at the time of the table export. All SCN per tables are written to a file named TABLES_SCN.log by default, it can be changed using -C | --cdc_file option. These SCN registered per table during COPY or INSERT export can be used with a CDC tool. The format of the file is tablename:SCN per line. =head2 Importing BLOB as large objects By default Ora2Pg imports Oracle BLOB as bytea, the destination column is created using the bytea data type. If you want to use large object instead of bytea, just add the --blob_to_lo option to the ora2pg command. It will create the destination column as data type Oid and will save the BLOB as a large object using the lo_from_bytea() function. The Oid returned by the call to lo_from_bytea() is inserted in the destination column instead of a bytea. Because of the use of the function this option can only be used with actions SHOW_COLUMN, TABLE and INSERT. Action COPY is not allowed. If you want to use COPY or have huge size BLOB ( > 1GB) than can not be imported using lo_from_bytea() you can add option --lo_import to the ora2pg command. This will allow to import data in two passes. 1) Export data using COPY or INSERT will set the Oid destination column for BLOB to value 0 and save the BLOB value into a dedicated file. It will also create a Shell script to import the BLOB files into the database using psql command \lo_import and to update the table Oid column to the returned large object Oid. The script is named lo_import-TABLENAME.sh 2) Execute all scripts lo_import-TABLENAME.sh after setting the environment variables PGDATABASE and optionally PGHOST, PGPORT, PGUSER, etc. if they do not correspond to the default values for libpq. You might also execute manually a VACUUM FULL on the table to remove the bloat created by the table update. Limitation: the table must have a primary key, it is used to set the WHERE clause to update the Oid column after the large object import. Importing BLOB using this second method (--lo_import) is very slow so it should be reserved to rows where the BLOB > 1GB for all other rows use the option --blob_to_lo. To filter the rows you can use the WHERE configuration directive in ora2pg.conf. =head1 SUPPORT =head2 Author / Maintainer Gilles Darold Please report any bugs, patches, help, etc. to . =head2 Feature request If you need new features let me know at . This help a lot to develop a better/useful tool. =head2 How to contribute ? Any contribution to build a better tool is welcome, you just have to send me your ideas, features request or patches and there will be applied. =head1 LICENSE Copyright (c) 2000-2025 Gilles Darold - All rights reserved. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see < http://www.gnu.org/licenses/ >. =head1 ACKNOWLEDGEMENT I must thanks a lot all the great contributors, see changelog for all acknowledgments. ora2pg-25.0/doc/ora2pg.3000066400000000000000000005451101500113072400146540ustar00rootroot00000000000000.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.42) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "ORA2PG 1" .TH ORA2PG 1 "2024-08-07" "perl v5.34.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" Ora2Pg \- Oracle to PostgreSQL database schema converter .SH "DESCRIPTION" .IX Header "DESCRIPTION" Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scans it automatically and extracts its structure or data, then generates \s-1SQL\s0 scripts that you can load into your PostgreSQL database. .PP Ora2Pg can be used for anything from reverse engineering Oracle database to huge enterprise database migration or simply replicating some Oracle data into a PostgreSQL database. It is really easy to use and doesn't require any Oracle database knowledge other than providing the parameters needed to connect to the Oracle database. .SH "FEATURES" .IX Header "FEATURES" Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), the only thing you have to modify is the configuration file ora2pg.conf by setting the \s-1DSN\s0 to the Oracle database and optionally the name of a schema. Once that's done you just have to set the type of export you want: \s-1TABLE\s0 with constraints, \&\s-1VIEW, MVIEW, TABLESPACE, SEQUENCE, INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, PACKAGE, PARTITION, TYPE, INSERT\s0 or \s-1COPY, FDW, QUERY, KETTLE, SYNONYM.\s0 .PP By default Ora2Pg exports to a file that you can load into PostgreSQL with the psql client, but you can also import directly into a PostgreSQL database by setting its \s-1DSN\s0 into the configuration file. With all configuration options of ora2pg.conf you have full control of what should be exported and how. .PP Features included: .PP .Vb 10 \& \- Export full database schema (tables, views, sequences, indexes), with \& unique, primary, foreign key and check constraints. \& \- Export grants/privileges for users and groups. \& \- Export range/list partitions and sub partitions. \& \- Export a table selection (by specifying the table names). \& \- Export Oracle schema to a PostgreSQL 8.4+ schema. \& \- Export predefined functions, triggers, procedures, packages and \& package bodies. \& \- Export full data or following a WHERE clause. \& \- Full support of Oracle BLOB object as PG BYTEA. \& \- Export Oracle views as PG tables. \& \- Export Oracle user defined types. \& \- Provide some basic automatic conversion of PLSQL code to PLPGSQL. \& \- Works on any platform. \& \- Export Oracle tables as foreign data wrapper tables. \& \- Export materialized view. \& \- Show a report of an Oracle database content. \& \- Migration cost assessment of an Oracle database. \& \- Migration difficulty level assessment of an Oracle database. \& \- Migration cost assessment of PL/SQL code from a file. \& \- Migration cost assessment of Oracle SQL queries stored in a file. \& \- Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) \& \- Export Oracle locator and spatial geometries into PostGis. \& \- Export DBLINK as Oracle FDW. \& \- Export SYNONYMS as views. \& \- Export DIRECTORY as external table or directory for external_file extension. \& \- Dispatch a list of SQL orders over multiple PostgreSQL connections \& \- Perform a diff between Oracle and PostgreSQL database for test purpose. \& \- MySQL/MariaDB and Microsoft SQL Server migration. .Ve .PP Ora2Pg does its best to automatically convert your Oracle database to PostgreSQL but there's still manual works to do. The Oracle specific \s-1PL/SQL\s0 code generated for functions, procedures, packages and triggers has to be reviewed to match the PostgreSQL syntax. You will find some useful recommendations on porting Oracle \s-1PL/SQL\s0 code to PostgreSQL \s-1PL/PGSQL\s0 at \*(L"Converting from other Databases to PostgreSQL\*(R", section: Oracle (http://wiki.postgresql.org/wiki/Main_Page). .PP See http://ora2pg.darold.net/report.html for a \s-1HTML\s0 sample of an Oracle database migration report. .SH "INSTALLATION" .IX Header "INSTALLATION" All Perl modules can always be found at \s-1CPAN\s0 (http://search.cpan.org/). Just type the full name of the module (ex: DBD::Oracle) into the search input box, it will brings you the page for download. .PP Releases of Ora2Pg stay at \s-1SF\s0.net (https://sourceforge.net/projects/ora2pg/). .PP Under Windows you should install Strawberry Perl (http://strawberryperl.com/) and the OSes corresponding Oracle clients. Since version 5.32 this Perl distribution include pre-compiled driver of DBD::Oracle and DBD::Pg. .SS "Requirement" .IX Subsection "Requirement" The Oracle Instant Client or a full Oracle installation must be installed on the system. You can download the \s-1RPM\s0 from Oracle download center: .PP .Vb 4 \& rpm \-ivh oracle\-instantclient12.2\-basic\-12.2.0.1.0\-1.x86_64.rpm \& rpm \-ivh oracle\-instantclient12.2\-devel\-12.2.0.1.0\-1.x86_64.rpm \& rpm \-ivh oracle\-instantclient12.2\-jdbc\-12.2.0.1.0\-1.x86_64.rpm \& rpm \-ivh oracle\-instantclient12.2\-sqlplus\-12.2.0.1.0\-1.x86_64.rpm .Ve .PP or simply download the corresponding \s-1ZIP\s0 archives from Oracle download center and install them where you want, for example: /opt/oracle/instantclient_12_2/ .PP You also need a modern Perl distribution (perl 5.10 and more). To connect to a database and proceed to his migration you need the \s-1DBI\s0 Perl module > 1.614. To migrate an Oracle database you need the DBD::Oracle Perl modules to be installed. .PP To install DBD::Oracle and have it working you need to have the Oracle client libraries installed and the \s-1ORACLE_HOME\s0 environment variable must be defined. .PP If you plan to export a MySQL database you need to install the Perl module DBD::MySQL which requires that the mysql client libraries are installed. .PP If you plan to export a \s-1SQL\s0 Server database you need to install the Perl module \&\s-1DBD::ODBC\s0 which requires that the unixODBC package is installed. .PP On some Perl distribution you may need to install the Time::HiRes Perl module. .PP If your distribution doesn't include these Perl modules you can install them using \s-1CPAN:\s0 .PP .Vb 4 \& perl \-MCPAN \-e \*(Aqinstall DBD::Oracle\*(Aq \& perl \-MCPAN \-e \*(Aqinstall DBD::MySQL\*(Aq \& perl \-MCPAN \-e \*(Aqinstall DBD::ODBC\*(Aq \& perl \-MCPAN \-e \*(Aqinstall Time::HiRes\*(Aq .Ve .PP otherwise use the packages provided by your distribution. .SS "Optional" .IX Subsection "Optional" By default Ora2Pg dumps export to flat files, to load them into your PostgreSQL database you need the PostgreSQL client (psql). If you don't have it on the host running Ora2Pg you can always transfer these files to a host with the psql client installed. If you prefer to load export 'on the fly', the perl module DBD::Pg is required. .PP Ora2Pg allows you to dump all output in a compressed gzip file, to do that you need the Compress::Zlib Perl module or if you prefer using bzip2 compression, the program bzip2 must be available in your \s-1PATH.\s0 .PP If your distribution doesn't include these Perl modules you can install them using \s-1CPAN:\s0 .PP .Vb 2 \& perl \-MCPAN \-e \*(Aqinstall DBD::Pg\*(Aq \& perl \-MCPAN \-e \*(Aqinstall Compress::Zlib\*(Aq .Ve .PP otherwise use the packages provided by your distribution. .SS "Instruction for \s-1SQL\s0 Server" .IX Subsection "Instruction for SQL Server" For \s-1SQL\s0 Server you need to install the unixodbc package and the Perl \&\s-1DBD::ODBC\s0 driver: .PP .Vb 2 \& sudo apt install unixodbc \& sudo apt install libdbd\-odbc\-perl .Ve .PP or .PP .Vb 3 \& sudo yum install unixodbc \& sudo yum install perl\-DBD\-ODBC \& sudo yum install perl\-DBD\-Pg .Ve .PP then install the Microsoft \s-1ODBC\s0 Driver for \s-1SQL\s0 Server. Follow the instructions relative to your operating system from here: .PP .Vb 1 \& https://docs.microsoft.com/fr\-fr/sql/connect/odbc/linux\-mac/installing\-the\-microsoft\-odbc\-driver\-for\-sql\-server?view=sql\-server\-ver16 .Ve .PP Once it is done set the following in the /etc/odbcinst.ini file by adjusting the \s-1SQL\s0 Server \s-1ODBC\s0 driver version: .PP .Vb 4 \& [msodbcsql18] \& Description=Microsoft ODBC Driver 18 for SQL Server \& Driver=/opt/microsoft/msodbcsql18/lib64/libmsodbcsql\-18.0.so.1.1 \& UsageCount=1 .Ve .PP See \s-1ORACLE_DSN\s0 to know how to use the driver to connect to your \s-1MSSQL\s0 database. .SS "Installing Ora2Pg" .IX Subsection "Installing Ora2Pg" Like any other Perl Module Ora2Pg can be installed with the following commands: .PP .Vb 4 \& tar xjf ora2pg\-x.x.tar.bz2 \& cd ora2pg\-x.x/ \& perl Makefile.PL \& make && make install .Ve .PP This will install Ora2Pg.pm into your site Perl repository, ora2pg into /usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. .PP On Windows(tm) OSes you may use instead: .PP .Vb 2 \& perl Makefile.PL \& gmake && gmake install .Ve .PP This will install scripts and libraries into your Perl site installation directory and the ora2pg.conf file as well as all documentation files into C:\eora2pg\e .PP To install ora2pg in a different directory than the default one, simply use this command: .PP .Vb 2 \& perl Makefile.PL PREFIX= \& make && make install .Ve .PP then set \s-1PERL5LIB\s0 to the path to your installation directory before using Ora2Pg. .PP .Vb 2 \& export PERL5LIB= \& ora2pg \-c config/ora2pg.conf \-t TABLE \-b outdir/ .Ve .SS "Packaging" .IX Subsection "Packaging" If you want to build the binary package for your preferred Linux distribution take a look at the packaging/ directory of the source tarball. There is everything to build \s-1RPM,\s0 Slackware and Debian packages. See \s-1README\s0 file in that directory. .SS "Installing DBD::Oracle" .IX Subsection "Installing DBD::Oracle" Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle database from perl \s-1DBI.\s0 To get DBD::Oracle get it from \s-1CPAN\s0 a perl module repository. .PP After setting \s-1ORACLE_HOME\s0 and \s-1LD_LIBRARY_PATH\s0 environment variables as root user, install DBD::Oracle. Proceed as follow: .PP .Vb 3 \& export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib \& export ORACLE_HOME=/usr/lib/oracle/12.2/client64 \& perl \-MCPAN \-e \*(Aqinstall DBD::Oracle\*(Aq .Ve .PP If you are running for the first time it will ask many questions; you can keep defaults by pressing \s-1ENTER\s0 key, but you need to give one appropriate mirror site for \s-1CPAN\s0 to download the modules. Install through \s-1CPAN\s0 manually if the above doesn't work: .PP .Vb 9 \& #perl \-MCPAN \-e shell \& cpan> get DBD::Oracle \& cpan> quit \& cd ~/.cpan/build/DBD\-Oracle* \& export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib \& export ORACLE_HOME=/usr/lib/oracle/11.2/client64 \& perl Makefile.PL \& make \& make install .Ve .PP Installing DBD::Oracle require that the three Oracle packages: instant-client, \&\s-1SDK\s0 and SQLplus are installed as well as the libaio1 library. .PP If you are using Instant Client from \s-1ZIP\s0 archives, the \s-1LD_LIBRARY_PATH\s0 and \&\s-1ORACLE_HOME\s0 will be the same and must be set to the directory where you have installed the files. For example: /opt/oracle/instantclient_12_2/ .SH "CONFIGURATION" .IX Header "CONFIGURATION" Ora2Pg configuration can be as simple as choosing the Oracle database to export and choose the export type. This can be done in a minute. .PP By reading this documentation you will also be able to: .PP .Vb 6 \& \- Select only certain tables and/or column for export. \& \- Rename some tables and/or column during export. \& \- Select data to export following a WHERE clause per table. \& \- Delay database constraints during data loading. \& \- Compress exported data to save disk space. \& \- and much more. .Ve .PP The full control of the Oracle database migration is taken though a single configuration file named ora2pg.conf. The format of this file consist in a directive name in upper case followed by tab character and a value. Comments are lines beginning with a #. .PP There's no specific order to place the configuration directives, they are set at the time they are read in the configuration file. .PP For configuration directives that just take a single value, you can use them multiple time in the configuration file but only the last occurrence found in the file will be used. For configuration directives that allow a list of value, you can use it multiple time, the values will be appended to the list. If you use the \s-1IMPORT\s0 directive to load a custom configuration file, directives defined in this file will be stores from the place the \s-1IMPORT\s0 directive is found, so it is better to put it at the end of the configuration file. .PP Values set in command line options will override values from the configuration file. .SS "Ora2Pg usage" .IX Subsection "Ora2Pg usage" First of all be sure that libraries and binaries path include the Oracle Instant Client installation: .PP .Vb 2 \& export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib \& export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" .Ve .PP By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration file, if the file exist you can simply execute: .PP .Vb 1 \& /usr/local/bin/ora2pg .Ve .PP or under Windows(tm) run ora2pg.bat file, located in your perl bin directory. Windows(tm) users may also find a template configuration file in C:\eora2pg .PP If you want to call another configuration file, just give the path as command line argument: .PP .Vb 1 \& /usr/local/bin/ora2pg \-c /etc/ora2pg/new_ora2pg.conf .Ve .PP Here are all command line parameters available when using ora2pg: .PP Usage: ora2pg [\-dhpqv \-\-estimate_cost \-\-dump_as_html] [\-\-option value] .PP .Vb 10 \& \-a | \-\-allow str : Comma separated list of objects to allow from export. \& Can be used with SHOW_COLUMN too. \& \-b | \-\-basedir dir: Set the default output directory, where files \& resulting from exports will be stored. \& \-c | \-\-conf file : Set an alternate configuration file other than the \& default /etc/ora2pg/ora2pg.conf. \& \-C | \-\-cdc_file file: File used to store/read SCN per table during export. \& default: TABLES_SCN.log in the current directory. This \& is the file written by the \-\-cdc_ready option. \& \-d | \-\-debug : Enable verbose output. \& \-D | \-\-data_type str : Allow custom type replacement at command line. \& \-e | \-\-exclude str: Comma separated list of objects to exclude from export. \& Can be used with SHOW_COLUMN too. \& \-h | \-\-help : Print this short help. \& \-g | \-\-grant_object type : Extract privilege from the given object type. \& See possible values with GRANT_OBJECT configuration. \& \-i | \-\-input file : File containing Oracle PL/SQL code to convert with \& no Oracle database connection initiated. \& \-j | \-\-jobs num : Number of parallel process to send data to PostgreSQL. \& \-J | \-\-copies num : Number of parallel connections to extract data from Oracle. \& \-l | \-\-log file : Set a log file. Default is stdout. \& \-L | \-\-limit num : Number of tuples extracted from Oracle and stored in \& memory before writing, default: 10000. \& \-m | \-\-mysql : Export a MySQL database instead of an Oracle schema. \& \-M | \-\-mssql : Export a Microsoft SQL Server database. \& \-n | \-\-namespace schema : Set the Oracle schema to extract from. \& \-N | \-\-pg_schema schema : Set PostgreSQL\*(Aqs search_path. \& \-o | \-\-out file : Set the path to the output file where SQL will \& be written. Default: output.sql in running directory. \& \-O | \-\-options : Used to override any configuration parameter, it can \& be used multiple time. Syntax: \-O "PARAM_NAME=value" \& \-p | \-\-plsql : Enable PLSQL to PLPGSQL code conversion. \& \-P | \-\-parallel num: Number of parallel tables to extract at the same time. \& \-q | \-\-quiet : Disable progress bar. \& \-r | \-\-relative : use \eir instead of \ei in the psql scripts generated. \& \-s | \-\-source DSN : Allow to set the Oracle DBI datasource. \& \-S | \-\-scn SCN : Allow to set the Oracle System Change Number (SCN) to \& use to export data. It will be used in the WHERE clause \& to get the data. It is used with action COPY or INSERT. \& \-t | \-\-type export: Set the export type. It will override the one \& given in the configuration file (TYPE). \& \-T | \-\-temp_dir dir: Set a distinct temporary directory when two \& or more ora2pg are run in parallel. \& \-u | \-\-user name : Set the Oracle database connection user. \& ORA2PG_USER environment variable can be used instead. \& \-v | \-\-version : Show Ora2Pg Version and exit. \& \-w | \-\-password pwd : Set the password of the Oracle database user. \& ORA2PG_PASSWD environment variable can be used instead. \& \-W | \-\-where clause : Set the WHERE clause to apply to the Oracle query to \& retrieve data. Can be used multiple time. \& \-\-forceowner : Force ora2pg to set tables and sequences owner like in \& Oracle database. If the value is set to a username this one \& will be used as the objects owner. By default it\*(Aqs the user \& used to connect to the Pg database that will be the owner. \& \-\-nls_lang code: Set the Oracle NLS_LANG client encoding. \& \-\-client_encoding code: Set the PostgreSQL client encoding. \& \-\-view_as_table str: Comma separated list of views to export as table. \& \-\-estimate_cost : Activate the migration cost evaluation with SHOW_REPORT \& \-\-cost_unit_value minutes: Number of minutes for a cost evaluation unit. \& default: 5 minutes, corresponds to a migration conducted by a \& PostgreSQL expert. Set it to 10 if this is your first migration. \& \-\-dump_as_html : Force ora2pg to dump report in HTML, used only with \& SHOW_REPORT. Default is to dump report as simple text. \& \-\-dump_as_csv : As above but force ora2pg to dump report in CSV. \& \-\-dump_as_json : As above but force ora2pg to dump report in JSON. \& \-\-dump_as_sheet : Report migration assessment with one CSV line per database. \& \-\-dump_as_file_prefix : Filename prefix, suffix will be added depending on \& dump_as_* selected switches, suffixes \& will be .html, .csv, .json. \& \-\-init_project name: Initialise a typical ora2pg project tree. Top directory \& will be created under project base dir. \& \-\-project_base dir : Define the base dir for ora2pg project trees. Default \& is current directory. \& \-\-print_header : Used with \-\-dump_as_sheet to print the CSV header \& especially for the first run of ora2pg. \& \-\-human_days_limit num : Set the number of person\-days limit where the migration \& assessment level switch from B to C. Default is set to \& 5 person\-days. \& \-\-audit_user list : Comma separated list of usernames to filter queries in \& the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT \& and QUERY export type. \& \-\-pg_dsn DSN : Set the datasource to PostgreSQL for direct import. \& \-\-pg_user name : Set the PostgreSQL user to use. \& \-\-pg_pwd password : Set the PostgreSQL password to use. \& \-\-count_rows : Force ora2pg to perform a real row count in TEST, \& TEST_COUNT and SHOW_TABLE actions. \& \-\-no_header : Do not append Ora2Pg header to output file \& \-\-oracle_speed : Use to know at which speed Oracle is able to send \& data. No data will be processed or written. \& \-\-ora2pg_speed : Use to know at which speed Ora2Pg is able to send \& transformed data. Nothing will be written. \& \-\-blob_to_lo : export BLOB as large objects, can only be used with \& action SHOW_COLUMN, TABLE and INSERT. \& \-\-cdc_ready : use current SCN per table to export data and register \& them into a file named TABLES_SCN.log per default. It \& can be changed using \-C | \-\-cdc_file. \& \-\-lo_import : use psql \elo_import command to import BLOB as large \& object. Can be use to import data with COPY and import \& large object manually in a second pass. It is recquired \& for BLOB > 1GB. See documentation for more explanation. \& \-\-mview_as_table str: Comma separated list of materialized views to export \& as regular table. \& \-\-drop_if_exists : Drop the object before creation if it exists. \& \-\-delete clause : Set the DELETE clause to apply to the Oracle query to \& be applied before importing data. Can be used multiple \& time. \& \-\-oracle_fdw_prefetch: Set the oracle_fdw prefetch value. Larger values \& generally result in faster data transfer at the cost \& of greater memory utilisation at the destination. .Ve .PP See full documentation at https://ora2pg.darold.net/ for more help or see manpage with 'man ora2pg'. .PP ora2pg will return 0 on success, 1 on error. It will return 2 when a child process has been interrupted and you've gotten the warning message: \*(L"\s-1WARNING:\s0 an error occurs during data export. Please check what's happen.\*(R" Most of the time this is an \s-1OOM\s0 issue, first try reducing \s-1DATA_LIMIT\s0 value. .PP For developers, it is possible to add your own custom option(s) in the Perl script ora2pg as any configuration directive from ora2pg.conf can be passed in lower case to the new Ora2Pg object instance. See ora2pg code on how to add your own option. .PP Note that performance might be improved by updating stats on oracle: .PP .Vb 5 \& BEGIN \& DBMS_STATS.GATHER_SCHEMA_STATS \& DBMS_STATS.GATHER_DATABASE_STATS \& DBMS_STATS.GATHER_DICTIONARY_STATS \& END; .Ve .SS "Generate a migration template" .IX Subsection "Generate a migration template" The two options \-\-project_base and \-\-init_project when used indicate to ora2pg that he has to create a project template with a work tree, a configuration file and a script to export all objects from the Oracle database. Here a sample of the command usage: .PP .Vb 10 \& ora2pg \-\-project_base /app/migration/ \-\-init_project test_project \& Creating project test_project. \& /app/migration/test_project/ \& schema/ \& dblinks/ \& directories/ \& functions/ \& grants/ \& mviews/ \& packages/ \& partitions/ \& procedures/ \& sequences/ \& synonyms/ \& tables/ \& tablespaces/ \& triggers/ \& types/ \& views/ \& sources/ \& functions/ \& mviews/ \& packages/ \& partitions/ \& procedures/ \& triggers/ \& types/ \& views/ \& data/ \& config/ \& reports/ \& \& Generating generic configuration file \& Creating script export_schema.sh to automate all exports. \& Creating script import_all.sh to automate all imports. .Ve .PP It create a generic config file where you just have to define the Oracle database connection and a shell script called export_schema.sh. The sources/ directory will contains the Oracle code, the schema/ will contains the code ported to PostgreSQL. The reports/ directory will contains the html reports with the migration cost assessment. .PP If you want to use your own default config file, use the \-c option to give the path to that file. Rename it with .dist suffix if you want ora2pg to apply the generic configuration values otherwise, the configuration file will be copied untouched. .PP Once you have set the connection to the Oracle Database you can execute the script export_schema.sh that will export all object type from your Oracle database and output \s-1DDL\s0 files into the schema's subdirectories. At end of the export it will give you the command to export data later when the import of the schema will be done and verified. .PP You can choose to load the \s-1DDL\s0 files generated manually or use the second script import_all.sh to import those file interactively. If this kind of migration is not something current for you it's recommended you to use those scripts. .SS "Oracle database connection" .IX Subsection "Oracle database connection" There's 5 configuration directives to control the access to the Oracle database. .IP "\s-1ORACLE_HOME\s0" 4 .IX Item "ORACLE_HOME" Used to set \s-1ORACLE_HOME\s0 environment variable to the Oracle libraries required by the DBD::Oracle Perl module. .IP "\s-1ORACLE_DSN\s0" 4 .IX Item "ORACLE_DSN" This directive is used to set the data source name in the form standard \s-1DBI DSN.\s0 For example: .Sp .Vb 1 \& dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 .Ve .Sp or .Sp .Vb 1 \& dbi:Oracle:DB_SID .Ve .Sp On 18c this could be for example: .Sp .Vb 1 \& dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 .Ve .Sp for the second notation the \s-1SID\s0 should be declared in the well known file \&\f(CW$ORACLE_HOME\fR/network/admin/tnsnames.ora or in the path given to the \s-1TNS_ADMIN\s0 environment variable. .Sp For MySQL the \s-1DSN\s0 will lool like this: .Sp .Vb 1 \& dbi:mysql:host=192.168.1.10;database=sakila;port=3306 .Ve .Sp the 'sid' part is replaced by 'database'. .Sp For \s-1MS SQL\s0 Server it will look like this: .Sp .Vb 1 \& dbi:ODBC:driver=msodbcsql18;server=mydb.database.windows.net;database=testdb;TrustServerCertificate=yes .Ve .IP "\s-1ORACLE_USER\s0 et \s-1ORACLE_PWD\s0" 4 .IX Item "ORACLE_USER et ORACLE_PWD" These two directives are used to define the user and password for the Oracle database connection. Note that if you can it is better to login as Oracle super admin to avoid grants problem during the database scan and be sure that nothing is missing. .Sp If you do not supply a credential with \s-1ORACLE_PWD\s0 and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If \&\s-1ORACLE_USER\s0 is not set it will be asked interactively too. .Sp To connect to a local \s-1ORACLE\s0 instance with connections \*(L"as sysdba\*(R" you have to set \s-1ORACLE_USER\s0 to \*(L"/\*(R" and an empty password. .Sp To make a connection using an Oracle Secure External Password Store (\s-1SEPS\s0), first configure the Oracle Wallet and then set both the \s-1ORACLE_USER\s0 and \&\s-1ORACLE_PWD\s0 directives to the special value of \*(L"_\|_SEPS_\|_\*(R" (without the quotes but with the double underscore). .IP "\s-1USER_GRANTS\s0" 4 .IX Item "USER_GRANTS" Set this directive to 1 if you connect the Oracle database as simple user and do not have enough grants to extract things from the \s-1DBA_...\s0 tables. It will use tables \s-1ALL_...\s0 instead. .Sp Warning: if you use export type \s-1GRANT,\s0 you must set this configuration option to 0 or it will not work. .IP "\s-1TRANSACTION\s0" 4 .IX Item "TRANSACTION" This directive may be used if you want to change the default isolation level of the data export transaction. Default is now to set the level to a serializable transaction to ensure data consistency. The allowed values for this directive are: .Sp .Vb 4 \& readonly: \*(AqSET TRANSACTION READ ONLY\*(Aq, \& readwrite: \*(AqSET TRANSACTION READ WRITE\*(Aq, \& serializable: \*(AqSET TRANSACTION ISOLATION LEVEL SERIALIZABLE\*(Aq \& committed: \*(AqSET TRANSACTION ISOLATION LEVEL READ COMMITTED\*(Aq, .Ve .Sp Releases before 6.2 used to set the isolation level to \s-1READ ONLY\s0 transaction but in some case this was breaking data consistency so now default is set to \&\s-1SERIALIZABLE.\s0 .IP "\s-1INPUT_FILE\s0" 4 .IX Item "INPUT_FILE" This directive did not control the Oracle database connection or unless it purely disables the use of any Oracle database by accepting a file as argument. Set this directive to a file containing \s-1PL/SQL\s0 Oracle Code like function, procedure or full package body to prevent Ora2Pg from connecting to an Oracle database and just apply his conversion tool to the content of the file. This can be used with the most of export types: \s-1TABLE, TRIGGER, PROCEDURE, VIEW, FUNCTION\s0 or \s-1PACKAGE,\s0 etc. .IP "\s-1ORA_INITIAL_COMMAND\s0" 4 .IX Item "ORA_INITIAL_COMMAND" This directive can be used to send an initial command to Oracle, just after the connection. For example to unlock a policy before reading objects or to set some session parameters. This directive can be used multiple times. .SS "Data encryption with Oracle server" .IX Subsection "Data encryption with Oracle server" If your Oracle Client config file already includes the encryption method, then DBD:Oracle uses those settings to encrypt the connection while you extract the data. For example if you have configured the Oracle Client config file (sqlnet.ora or .sqlnet) with the following information: .PP .Vb 4 \& # Configure encryption of connections to Oracle \& SQLNET.ENCRYPTION_CLIENT = required \& SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) \& SQLNET.CRYPTO_SEED = \*(Aqshould be 10\-70 random characters\*(Aq .Ve .PP Any tool that uses the Oracle client to talk to the database will be encrypted if you setup session encryption like above. .PP For example, Perl's \s-1DBI\s0 uses DBD-Oracle, which uses the Oracle client for actually handling database communication. If the installation of Oracle client used by Perl is setup to request encrypted connections, then your Perl connection to an Oracle database will also be encrypted. .PP Full details at https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 .SS "Testing connection" .IX Subsection "Testing connection" Once you have set the Oracle database \s-1DSN\s0 you can execute ora2pg to see if it works: .PP .Vb 1 \& ora2pg \-t SHOW_VERSION \-c config/ora2pg.conf .Ve .PP will show the Oracle database server version. Take some time here to test your installation as most problems take place here, the other configuration steps are more technical. .SS "Troubleshooting" .IX Subsection "Troubleshooting" If the output.sql file has not exported anything other than the Pg transaction header and footer there's two possible reasons. The perl script ora2pg dump an ORA-XXX error, that mean that your \s-1DSN\s0 or login information are wrong, check the error and your settings and try again. The perl script says nothing and the output file is empty: the user lacks permission to extract something from the database. Try to connect to Oracle as super user or take a look at directive \&\s-1USER_GRANTS\s0 above and at next section, especially the \s-1SCHEMA\s0 directive. .IP "\s-1LOGFILE\s0" 4 .IX Item "LOGFILE" By default all messages are sent to the standard output. If you give a file path to that directive, all output will be appended to this file. .SS "Oracle schema to export" .IX Subsection "Oracle schema to export" The Oracle database export can be limited to a specific Schema or Namespace, this can be mandatory following the database connection user. .IP "\s-1SCHEMA\s0" 4 .IX Item "SCHEMA" This directive is used to set the schema name to use during export. For example: .Sp .Vb 1 \& SCHEMA APPS .Ve .Sp will extract objects associated to the \s-1APPS\s0 schema. .Sp When no schema name is provided and \s-1EXPORT_SCHEMA\s0 is enabled, Ora2Pg will export all objects from all schema of the Oracle instance with their names prefixed with the schema name. .IP "\s-1EXPORT_SCHEMA\s0" 4 .IX Item "EXPORT_SCHEMA" By default the Oracle schema is not exported into the PostgreSQL database and all objects are created under the default Pg namespace. If you want to also export this schema and create all objects under this namespace, set the \&\s-1EXPORT_SCHEMA\s0 directive to 1. This will set the schema search_path at top of export \s-1SQL\s0 file to the schema name set in the \s-1SCHEMA\s0 directive with the default pg_catalog schema. If you want to change this path, use the directive \s-1PG_SCHEMA.\s0 .IP "\s-1CREATE_SCHEMA\s0" 4 .IX Item "CREATE_SCHEMA" Enable/disable the \s-1CREATE SCHEMA SQL\s0 order at starting of the output file. It is enable by default and concern on \s-1TABLE\s0 export type. .IP "\s-1COMPILE_SCHEMA\s0" 4 .IX Item "COMPILE_SCHEMA" By default Ora2Pg will only export valid \s-1PL/SQL\s0 code. You can force Oracle to compile again the invalidated code to get a chance to have it obtain the valid status and then be able to export it. .Sp Enable this directive to force Oracle to compile schema before exporting code. When this directive is enabled and \s-1SCHEMA\s0 is set to a specific schema name, only invalid objects in this schema will be recompiled. If \s-1SCHEMA\s0 is not set then all schema will be recompiled. To force recompile invalid object in a specific schema, set \s-1COMPILE_SCHEMA\s0 to the schema name you want to recompile. .Sp This will ask to Oracle to validate the \s-1PL/SQL\s0 that could have been invalidate after a export/import for example. The '\s-1VALID\s0' or '\s-1INVALID\s0' status applies to functions, procedures, packages and user defined types. It also concern disabled triggers. .IP "\s-1EXPORT_INVALID\s0" 4 .IX Item "EXPORT_INVALID" If the above configuration directive is not enough to validate your \s-1PL/SQL\s0 code enable this configuration directive to allow export of all \s-1PL/SQL\s0 code even if it is marked as invalid. The '\s-1VALID\s0' or '\s-1INVALID\s0' status applies to functions, procedures, packages, triggers and user defined types. .IP "\s-1PG_SCHEMA\s0" 4 .IX Item "PG_SCHEMA" Allow you to defined/force the PostgreSQL schema to use. By default if you set \&\s-1EXPORT_SCHEMA\s0 to 1 the PostgreSQL search_path will be set to the schema name exported set as value of the \s-1SCHEMA\s0 directive. .Sp The value can be a comma delimited list of schema name but not when using \s-1TABLE\s0 export type because in this case it will generate the \s-1CREATE SCHEMA\s0 statement and it doesn't support multiple schema name. For example, if you set \s-1PG_SCHEMA\s0 to something like \*(L"user_schema, public\*(R", the search path will be set like this: .Sp .Vb 1 \& SET search_path = user_schema, public; .Ve .Sp forcing the use of an other schema (here user_schema) than the one from Oracle schema set in the \s-1SCHEMA\s0 directive. .Sp You can also set the default search_path for the PostgreSQL user you are using to connect to the destination database by using: .Sp .Vb 1 \& ALTER ROLE username SET search_path TO user_schema, public; .Ve .Sp in this case you don't have to set \s-1PG_SCHEMA.\s0 .IP "\s-1SYSUSERS\s0" 4 .IX Item "SYSUSERS" Without explicit schema, Ora2Pg will export all objects that not belongs to system schema or role: .Sp .Vb 12 \& SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, \& OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, \& SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, \& WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, \& FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, \& SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, \& SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, \& DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, \& AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, \& OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, \& AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, \& DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER .Ve .Sp Following your Oracle installation you may have several other system role defined. To append these users to the schema exclusion list, just set the \&\s-1SYSUSERS\s0 configuration directive to a comma-separated list of system user to exclude. For example: .Sp .Vb 1 \& SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH .Ve .Sp will add users \s-1INTERNAL\s0 and \s-1SYSDBA\s0 to the schema exclusion list. .IP "\s-1FORCE_OWNER\s0" 4 .IX Item "FORCE_OWNER" By default the owner of the database objects is the one you're using to connect to PostgreSQL using the psql command. If you use an other user (postgres for example) you can force Ora2Pg to set the object owner to be the one used in the Oracle database by setting the directive to 1, or to a completely different username by setting the directive value to that username. .IP "\s-1FORCE_SECURITY_INVOKER\s0" 4 .IX Item "FORCE_SECURITY_INVOKER" Ora2Pg use the function's security privileges set in Oracle and it is often defined as \s-1SECURITY DEFINER.\s0 If you want to override those security privileges for all functions and use \s-1SECURITY DEFINER\s0 instead, enable this directive. .IP "\s-1USE_TABLESPACE\s0" 4 .IX Item "USE_TABLESPACE" When enabled this directive force ora2pg to export all tables, indexes constraint and indexes using the tablespace name defined in Oracle database. This works only with tablespace that are not \s-1TEMP, USERS\s0 and \s-1SYSTEM.\s0 .IP "\s-1WITH_OID\s0" 4 .IX Item "WITH_OID" Activating this directive will force Ora2Pg to add \s-1WITH\s0 (\s-1OIDS\s0) when creating tables or views as tables. Default is same as PostgreSQL, disabled. .IP "\s-1LOOK_FORWARD_FUNCTION\s0" 4 .IX Item "LOOK_FORWARD_FUNCTION" List of schema to get functions/procedures meta information that are used in the current schema export. When replacing call to function with \s-1OUT\s0 parameters, if a function is declared in an other package then the function call rewriting can not be done because Ora2Pg only knows about functions declared in the current schema. By setting a comma separated list of schema as value of this directive, Ora2Pg will look forward in these packages for all functions/procedures/packages declaration before proceeding to current schema export. .IP "\s-1NO_FUNCTION_METADATA\s0" 4 .IX Item "NO_FUNCTION_METADATA" Force Ora2Pg to not look for function declaration. Note that this will prevent Ora2Pg to rewrite function replacement call if needed. Do not enable it unless looking forward at function breaks other export. .SS "Export type" .IX Subsection "Export type" The export action is perform following a single configuration directive '\s-1TYPE\s0', some other add more control on what should be really exported. .IP "\s-1TYPE\s0" 4 .IX Item "TYPE" Here are the different values of the \s-1TYPE\s0 directive, default is \s-1TABLE:\s0 .Sp .Vb 10 \& \- TABLE: Extract all tables with indexes, primary keys, unique keys, \& foreign keys and check constraints. \& \- VIEW: Extract only views. \& \- GRANT: Extract roles converted to Pg groups, users and grants on all \& objects. \& \- SEQUENCE: Extract all sequence and their last position. \& \- TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). \& \- TRIGGER: Extract triggers defined following actions. \& \- FUNCTION: Extract functions. \& \- PROCEDURE: Extract procedures. \& \- PACKAGE: Extract packages and package bodies. \& \- INSERT: Extract data as INSERT statement. \& \- COPY: Extract data as COPY statement. \& \- PARTITION: Extract range and list Oracle partitions with subpartitions. \& \- TYPE: Extract user defined Oracle type. \& \- FDW: Export Oracle tables as foreign table for Oracle, MySQL and SQL Server FDW. \& \- MVIEW: Export materialized view. \& \- QUERY: Try to automatically convert Oracle SQL queries. \& \- KETTLE: Generate XML ktr template files to be used by Kettle. \& \- DBLINK: Generate oracle foreign data wrapper server to use as dblink. \& \- SYNONYM: Export Oracle\*(Aqs synonyms as views on other schema\*(Aqs objects. \& \- DIRECTORY: Export Oracle\*(Aqs directories as external_file extension objects. \& \- LOAD: Dispatch a list of queries over multiple PostgreSQl connections. \& \- TEST: perform a diff between Oracle and PostgreSQL database. \& \- TEST_COUNT: perform a row count diff between Oracle and PostgreSQL table. \& \- TEST_VIEW: perform a count on both side of number of rows returned by views. \& \- TEST_DATA: perform data validation check on rows at both sides. \& \- SEQUENCE_VALUES: export DDL to set the last values of sequences .Ve .Sp Only one type of export can be perform at the same time so the \s-1TYPE\s0 directive must be unique. If you have more than one only the last found in the file will be registered. .Sp Some export type can not or should not be load directly into the PostgreSQL database and still require little manual editing. This is the case for \s-1GRANT, TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, QUERY\s0 and \s-1PACKAGE\s0 export types especially if you have \s-1PLSQL\s0 code or Oracle specific \s-1SQL\s0 in it. .Sp For \s-1TABLESPACE\s0 you must ensure that file path exist on the system and for \&\s-1SYNONYM\s0 you may ensure that the object's owners and schemas correspond to the new PostgreSQL database design. .Sp Note that you can chained multiple export by giving to the \s-1TYPE\s0 directive a comma-separated list of export type, but in this case you must not use \s-1COPY\s0 or \s-1INSERT\s0 with other export type. .Sp Ora2Pg will convert Oracle partition using table inheritance, trigger and functions. See document at Pg site: http://www.postgresql.org/docs/current/interactive/ddl\-partitioning.html .Sp The \s-1TYPE\s0 export allow export of user defined Oracle type. If you don't use the \&\-\-plsql command line parameter it simply dump Oracle user type asis else Ora2Pg will try to convert it to PostgreSQL syntax. .Sp The \s-1KETTLE\s0 export type requires that the Oracle and PostgreSQL \s-1DNS\s0 are defined. .Sp Since Ora2Pg v8.1 there's three new export types: .Sp .Vb 7 \& SHOW_VERSION : display Oracle version \& SHOW_SCHEMA : display the list of schema available in the database. \& SHOW_TABLE : display the list of tables available. \& SHOW_COLUMN : display the list of tables columns available and the \& Ora2PG conversion type from Oracle to PostgreSQL that will be \& applied. It will also warn you if there\*(Aqs PostgreSQL reserved \& words in Oracle object names. .Ve .Sp Here is an example of the \s-1SHOW_COLUMN\s0 output: .Sp .Vb 11 \& [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: \*(AqCURRENT_SCHEMA\*(Aq is a reserved word in PostgreSQL) \& CONSTRAINT : NUMBER(22) => bigint (Warning: \*(AqCONSTRAINT\*(Aq is a reserved word in PostgreSQL) \& FREEZE : VARCHAR2(25) => varchar(25) (Warning: \*(AqFREEZE\*(Aq is a reserved word in PostgreSQL) \& ... \& [6] TABLE LOCATIONS (23 rows) \& LOCATION_ID : NUMBER(4) => smallint \& STREET_ADDRESS : VARCHAR2(40) => varchar(40) \& POSTAL_CODE : VARCHAR2(12) => varchar(12) \& CITY : VARCHAR2(30) => varchar(30) \& STATE_PROVINCE : VARCHAR2(25) => varchar(25) \& COUNTRY_ID : CHAR(2) => char(2) .Ve .Sp Those extraction keywords are use to only display the requested information and exit. This allows you to quickly know on what you are going to work. .Sp The \s-1SHOW_COLUMN\s0 allow an other ora2pg command line option: '\-\-allow relname' or '\-a relname' to limit the displayed information to the given table. .Sp The \s-1SHOW_ENCODING\s0 export type will display the \s-1NLS_LANG\s0 and \s-1CLIENT_ENCODING\s0 values that Ora2Pg will used and the real encoding of the Oracle database with the corresponding client encoding that could be used with PostgreSQL .Sp Ora2Pg allow you to export your Oracle, MySQL or \s-1MSSQL\s0 table definition to be use with the oracle_fdw, mysql_fdw or tds_fdw foreign data wrapper. By using type \s-1FDW\s0 your tables will be exported as follow: .Sp .Vb 5 \& CREATE FOREIGN TABLE oratab ( \& id integer NOT NULL, \& text character varying(30), \& floating double precision NOT NULL \& ) SERVER oradb OPTIONS (table \*(AqORATAB\*(Aq); .Ve .Sp Now you can use the table like a regular PostgreSQL table. .Sp Release 10 adds a new export type destined to evaluate the content of the database to migrate, in terms of objects and cost to end the migration: .Sp .Vb 1 \& SHOW_REPORT : show a detailed report of the Oracle database content. .Ve .Sp Here is a sample of report: http://ora2pg.darold.net/report.html .Sp There also a more advanced report with migration cost. See the dedicated chapter about Migration Cost Evaluation. .IP "\s-1ESTIMATE_COST\s0" 4 .IX Item "ESTIMATE_COST" Activate the migration cost evaluation. Must only be used with \s-1SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE\s0 and \s-1QUERY\s0 export type. Default is disabled. You may want to use the \-\-estimate_cost command line option instead to activate this functionality. Note that enabling this directive will force \s-1PLSQL_PGSQL\s0 activation. .IP "\s-1COST_UNIT_VALUE\s0" 4 .IX Item "COST_UNIT_VALUE" Set the value in minutes of the migration cost evaluation unit. Default is five minutes per unit. See \-\-cost_unit_value to change the unit value at command line. .IP "\s-1DUMP_AS_HTML\s0" 4 .IX Item "DUMP_AS_HTML" By default when using \s-1SHOW_REPORT\s0 the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in \s-1HTML\s0 format. .Sp See http://ora2pg.darold.net/report.html for a sample report. .IP "\s-1DUMP_AS_JSON\s0" 4 .IX Item "DUMP_AS_JSON" By default when using \s-1SHOW_REPORT\s0 the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in \s-1JSON\s0 format. .Sp See http://ora2pg.darold.net/report.html for a sample report. .IP "\s-1DUMP_AS_CSV\s0" 4 .IX Item "DUMP_AS_CSV" By default when using \s-1SHOW_REPORT\s0 the migration report is generated as simple text, enabling this directive will force ora2pg to create a report in \s-1CSV\s0 format. .Sp See http://ora2pg.darold.net/report.html for a sample report. .IP "\s-1DUMP_AS_FILE_PREFIX\s0" 4 .IX Item "DUMP_AS_FILE_PREFIX" By default when using \s-1SHOW_REPORT\s0 the migration report is generated to stout. Enabling this directive in conjunction with \s-1DUMP_AS_*\s0 directives will force ora2pg to create a report files with the given extensions and formats. This option allows you to combine multiple \s-1DUMP_AS_*\s0 formats. .Sp See http://ora2pg.darold.net/report.html for a sample report. .IP "\s-1HUMAN_DAYS_LIMIT\s0" 4 .IX Item "HUMAN_DAYS_LIMIT" Use this directive to redefined the number of person-days limit where the migration assessment level must switch from B to C. Default is set to 10 person-days. .IP "\s-1JOBS\s0" 4 .IX Item "JOBS" This configuration directive adds multiprocess support to \s-1COPY, FUNCTION\s0 and \s-1PROCEDURE\s0 export type, the value is the number of process to use. Default is multiprocess disable. .Sp This directive is used to set the number of cores to used to parallelize data import into PostgreSQL. During \s-1FUNCTION\s0 or \s-1PROCEDURE\s0 export type each function will be translated to plpgsql using a new process, the performances gain can be very important when you have tons of function to convert. .Sp There's no limitation in parallel processing than the number of cores and the PostgreSQL I/O performance capabilities. .Sp Doesn't work under Windows Operating System, it is simply disabled. .IP "\s-1ORACLE_COPIES\s0" 4 .IX Item "ORACLE_COPIES" This configuration directive adds multiprocess support to extract data from Oracle. The value is the number of process to use to parallelize the select query. Default is parallel query disable. .Sp The parallelism is built on splitting the query following of the number of cores given as value to \s-1ORACLE_COPIES\s0 as follow: .Sp .Vb 1 \& SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC .Ve .Sp where \s-1COLUMN\s0 is a technical key like a primary or unique key where split will be based and the current core used by the query (\s-1CUR_PROC\s0). You can also force the column name to use using the \s-1DEFINED_PK\s0 configuration directive. .Sp Doesn't work under Windows Operating System, it is simply disabled. .IP "\s-1DEFINED_PK\s0" 4 .IX Item "DEFINED_PK" This directive is used to defined the technical key to used to split the query between number of cores set with the \s-1ORACLE_COPIES\s0 variable. For example: .Sp .Vb 1 \& DEFINED_PK EMPLOYEES:employee_id .Ve .Sp The parallel query that will be used supposing that \-J or \s-1ORACLE_COPIES\s0 is set to 8: .Sp .Vb 1 \& SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N .Ve .Sp where N is the current process forked starting from 0. .IP "\s-1PARALLEL_TABLES\s0" 4 .IX Item "PARALLEL_TABLES" This directive is used to defined the number of tables that will be processed in parallel for data extraction. The limit is the number of cores on your machine. Ora2Pg will open one database connection for each parallel table extraction. This directive, when upper than 1, will invalidate \s-1ORACLE_COPIES\s0 but not \s-1JOBS,\s0 so the real number of process that will be used is \s-1PARALLEL_TABLES\s0 * \s-1JOBS.\s0 .Sp Note that this directive when set upper that 1 will also automatically enable the \s-1FILE_PER_TABLE\s0 directive if your are exporting to files. This is used to export tables and views in separate files. .Sp Use \s-1PARALLEL_TABLES\s0 to use parallelism with \s-1COPY, INSERT\s0 and \s-1TEST_DATA\s0 actions. It is also useful with \s-1TEST, TEST_COUNT,\s0 and \s-1SHOW_TABLE\s0 if \-\-count_rows is used for real row count. .IP "\s-1DEFAULT_PARALLELISM_DEGREE\s0" 4 .IX Item "DEFAULT_PARALLELISM_DEGREE" You can force Ora2Pg to use /*+ \s-1PARALLEL\s0(tbname, degree) */ hint in each query used to export data from Oracle by setting a value upper than 1 to this directive. A value of 0 or 1 disable the use of parallel hint. Default is disabled. .IP "\s-1FDW_SERVER\s0" 4 .IX Item "FDW_SERVER" This directive is used to set the name of the foreign data server that is used in the \*(L"\s-1CREATE SERVER\s0 name \s-1FOREIGN DATA WRAPPER\s0 ...\*(R" command. This name will then be used in the \*(L"\s-1CREATE FOREIGN TABLE ...\*(R" SQL\s0 commands and to import data using oracle_fdw. Default is no foreign server defined. This only concerns export type \s-1FDW, COPY\s0 and \s-1INSERT.\s0 For export type \s-1FDW\s0 the default value is orcl. .IP "\s-1FDW_IMPORT_SCHEMA\s0" 4 .IX Item "FDW_IMPORT_SCHEMA" Schema where foreign tables for data migration will be created. If you use several instances of ora2pg for data migration through the foreign data wrapper, you might need to change the name of the schema for each instance. Default: ora2pg_fdw_import .IP "\s-1ORACLE_FDW_PREFETCH\s0" 4 .IX Item "ORACLE_FDW_PREFETCH" The default behaviour of Ora2Pg is to \s-1NOT\s0 set the \*(L"prefetch\*(R" option for oracle_fdw when used for \s-1COPY\s0 and \s-1INSERT.\s0 This directive allows the prefetch to be set. See oracle_fdw documentation for the current default. .IP "\s-1ORACLE_FDW_COPY_MODE\s0" 4 .IX Item "ORACLE_FDW_COPY_MODE" When using Ora2Pg \s-1COPY\s0 with oracle_fdw it is possible to use two different modes: 1) \*(L"local\*(R", which uses psql on the host running Ora2Pg for the \*(L"\s-1TO\*(R"\s0 binary stream; 2) \*(L"server\*(R", which uses PostgreSQL server-side \s-1COPY\s0 for the \*(L"\s-1TO\*(R"\s0 binary stream. Both modes use psql for the \*(L"\s-1FROM STDIN BINARY\*(R".\s0 However, \&\*(L"local\*(R" runs the psql \*(L"\s-1FROM STDIN BINARY\*(R"\s0 on host Ora2Pg is run from, whereas \&\*(L"server\*(R" runs the psql \*(L"\s-1FROM STDIN BINARY\*(R"\s0 on the PostgreSQL server. \*(L"local\*(R" mode should work on any PostgreSQL-based system, including managed offerings, which are not expected to support use of \*(L"server\*(R" mode due to permissions. The default is \*(L"local\*(R" as this is compatible with more configurations. .IP "\s-1ORACLE_FDW_COPY_FORMAT\s0" 4 .IX Item "ORACLE_FDW_COPY_FORMAT" When using Ora2Pg \s-1COPY\s0 with oracle_fdw it is possible to use either \s-1BINARY\s0 or \&\s-1CSV\s0 data format. \s-1BINARY\s0 provides better performance, however, requires exact data type matching between the \s-1FDW\s0 and destination table. \s-1CSV\s0 provides greater flexibiliity with respect to data type matching: if the \s-1FDW\s0 and destination data types are functionally-compatible the columns can be copied. The default is \*(L"binary\*(R". .IP "\s-1DROP_FOREIGN_SCHEMA\s0" 4 .IX Item "DROP_FOREIGN_SCHEMA" By default Ora2Pg drops the temporary schema ora2pg_fdw_import used to import the Oracle foreign schema before each new import. If you want to preserve the existing schema because of modifications or the use of a third party server, disable this directive. .IP "\s-1EXTERNAL_TO_FDW\s0" 4 .IX Item "EXTERNAL_TO_FDW" This directive, enabled by default, allow to export Oracle's External Tables as file_fdw foreign tables. To not export these tables at all, set the directive to 0. .IP "\s-1INTERNAL_DATE_MAX\s0" 4 .IX Item "INTERNAL_DATE_MAX" Internal timestamp retrieves from custom type are extracted in the following format: 01\-JAN\-77 12.00.00.000000 \s-1AM.\s0 It is impossible to know the exact century that must be used, so by default any year below 49 will be added to 2000 and others to 1900. You can use this directive to change the default value 49. this is only relevant if you have user defined type with a column timestamp. .IP "\s-1AUDIT_USER\s0" 4 .IX Item "AUDIT_USER" Set the comma separated list of username that must be used to filter queries from the \s-1DBA_AUDIT_TRAIL\s0 table. Default is to not scan this table and to never look for queries. This parameter is used only with \&\s-1SHOW_REPORT\s0 and \s-1QUERY\s0 export type with no input file for queries. Note that queries will be normalized before output unlike when a file is given at input using the \-i option or \s-1INPUT\s0 directive. .IP "\s-1FUNCTION_CHECK\s0" 4 .IX Item "FUNCTION_CHECK" Disable this directive if you want to disable check_function_bodies. .Sp .Vb 1 \& SET check_function_bodies = false; .Ve .Sp It disables validation of the function body string during \s-1CREATE FUNCTION.\s0 Default is to use de postgresql.conf setting that enable it by default. .IP "\s-1ENABLE_BLOB_EXPORT\s0" 4 .IX Item "ENABLE_BLOB_EXPORT" Exporting \s-1BLOB\s0 takes time, in some circumstances you may want to export all data except the \s-1BLOB\s0 columns. In this case disable this directive and the \s-1BLOB\s0 columns will not be included into data export. Take care that the target bytea column do not have a \s-1NOT NULL\s0 constraint. .IP "\s-1ENABLE_CLOB_EXPORT\s0" 4 .IX Item "ENABLE_CLOB_EXPORT" Same behavior as \s-1ENABLE_BLOB_EXPORT\s0 but for \s-1CLOB.\s0 .IP "\s-1DATA_EXPORT_ORDER\s0" 4 .IX Item "DATA_EXPORT_ORDER" By default data export order will be done by sorting on table name. If you have huge tables at end of alphabetic order and you are using multiprocess, it can be better to set the sort order on size so that multiple small tables can be processed before the largest tables finish. In this case set this directive to size. Possible values are name and size. Note that export type \&\s-1SHOW_TABLE\s0 and \s-1SHOW_COLUMN\s0 will use this sort order too, not only \s-1COPY\s0 or \&\s-1INSERT\s0 export type. If you want to give you custom export order, just give a filename as value that contains the ordered list of table to export. Must be a list of one table per line, in uppercase for Oracle. .SS "Limiting objects to export" .IX Subsection "Limiting objects to export" You may want to export only a part of an Oracle database, here are a set of configuration directives that will allow you to control what parts of the database should be exported. .IP "\s-1ALLOW\s0" 4 .IX Item "ALLOW" This directive allows you to set a list of objects on which the export must be limited, excluding all other objects in the same type of export. The value is a space or comma-separated list of objects name to export. You can include valid regex into the list. For example: .Sp .Vb 1 \& ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ .Ve .Sp will export objects with name \s-1EMPLOYEES, COUNTRIES,\s0 all objects beginning with \&'\s-1SALE_\s0' and all objects with a name ending by '_GEOM_SEQ'. The object depends of the export type. Note that regex will not works with 8i database, you must use the % placeholder instead, Ora2Pg will use the \s-1LIKE\s0 operator. .Sp This is the manner to declare global filters that will be used with the current export type. You can also use extended filters that will be applied on specific objects or only on their related export type. For example: .Sp .Vb 1 \& ora2pg \-p \-c ora2pg.conf \-t TRIGGER \-a \*(AqTABLE[employees]\*(Aq .Ve .Sp will limit export of trigger to those defined on table employees. If you want to extract all triggers but not some \s-1INSTEAD OF\s0 triggers: .Sp .Vb 1 \& ora2pg \-c ora2pg.conf \-t TRIGGER \-e \*(AqVIEW[trg_view_.*]\*(Aq .Ve .Sp Or a more complex form: .Sp .Vb 2 \& ora2pg \-p \-c ora2pg.conf \-t TABLE \-a \*(AqTABLE[EMPLOYEES]\*(Aq \e \& \-e \*(AqINDEX[emp_.*];CKEY[emp_salary_min]\*(Aq .Ve .Sp This command will export the definition of the employee table but will exclude all index beginning with 'emp_' and the \s-1CHECK\s0 constraint called 'emp_salary_min'. .Sp When exporting partition you can exclude some partition tables by using .Sp .Vb 1 \& ora2pg \-p \-c ora2pg.conf \-t PARTITION \-e \*(AqPARTITION[PART_199.* PART_198.*]\*(Aq .Ve .Sp This will exclude partitioned tables for year 1980 to 1999 from the export but not the main partition table. The trigger will also be adapted to exclude those table. .Sp With \s-1GRANT\s0 export you can use this extended form to exclude some users from the export or limit the export to some others: .Sp .Vb 1 \& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqUSER1 USER2\*(Aq .Ve .Sp or .Sp .Vb 1 \& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqGRANT[USER1 USER2]\*(Aq .Ve .Sp will limit export grants to users \s-1USER1\s0 and \s-1USER2.\s0 But if you don't want to export grants on some functions for these users, for example: .Sp .Vb 1 \& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqUSER1 USER2\*(Aq \-e \*(AqFUNCTION[adm_.*];PROCEDURE[adm_.*]\*(Aq .Ve .Sp Advanced filters may need some learning. .Sp Oracle doesn't allow the use of lookahead expression so you may want to exclude some object that match the \s-1ALLOW\s0 regexp you have defined. For example if you want to export all table starting with E but not those starting with \s-1EXP\s0 it is not possible to do that in a single expression. This is why you can start a regular expression with the ! character to exclude object matching the regexp given just after. Our previous example can be written as follow: .Sp .Vb 1 \& ALLOW E.* !EXP.* .Ve .Sp it will be translated into: .Sp .Vb 1 \& REGEXP_LIKE(..., \*(Aq^E.*$\*(Aq) AND NOT REGEXP_LIKE(..., \*(Aq^EXP.*$\*(Aq) .Ve .Sp in the object search expression. .IP "\s-1EXCLUDE\s0" 4 .IX Item "EXCLUDE" This directive is the opposite of the previous, it allow you to define a space or comma-separated list of object name to exclude from the export. You can include valid regex into the list. For example: .Sp .Vb 1 \& EXCLUDE EMPLOYEES TMP_.* COUNTRIES .Ve .Sp will exclude object with name \s-1EMPLOYEES, COUNTRIES\s0 and all tables beginning with \&'tmp_'. .Sp For example, you can ban from export some unwanted function with this directive: .Sp .Vb 1 \& EXCLUDE write_to_.* send_mail_.* .Ve .Sp this example will exclude all functions, procedures or functions in a package with the name beginning with those regex. Note that regex will not work with 8i database, you must use the % placeholder instead, Ora2Pg will use the \s-1NOT LIKE\s0 operator. .Sp See above (directive '\s-1ALLOW\s0') for the extended syntax. .IP "\s-1NO_EXCLUDED_TABLE\s0" 4 .IX Item "NO_EXCLUDED_TABLE" By default Ora2Pg exclude from export some Oracle \*(L"garbage\*(R" tables that should never be part of an export. This behavior generates a lot of \s-1REGEXP_LIKE\s0 expressions which are slowing down the export when looking at tables. To disable this behavior enable this directive, you will have to exclude or clean up later by yourself the unwanted tables. The regexp used to exclude the table are defined in the array \f(CW@EXCLUDED_TABLES\fR in lib/Ora2Pg.pm. Note this is behavior is independant to the \s-1EXCLUDE\s0 configuration directive. .IP "\s-1VIEW_AS_TABLE\s0" 4 .IX Item "VIEW_AS_TABLE" Set which view to export as table. By default none. Value must be a list of view name or regexp separated by space or comma. If the object name is a view and the export type is \s-1TABLE,\s0 the view will be exported as a create table statement. If export type is \s-1COPY\s0 or \s-1INSERT,\s0 the corresponding data will be exported. .Sp See chapter \*(L"Exporting views as PostgreSQL table\*(R" for more details. .IP "\s-1MVIEW_AS_TABLE\s0" 4 .IX Item "MVIEW_AS_TABLE" Set which materialized view to export as table. By default none. Value must be a list of materialized view name or regexp separated by space or comma. If the object name is a materialized view and the export type is \s-1TABLE,\s0 the view will be exported as a create table statement. If export type is \s-1COPY\s0 or \s-1INSERT,\s0 the corresponding data will be exported. .IP "\s-1NO_VIEW_ORDERING\s0" 4 .IX Item "NO_VIEW_ORDERING" By default Ora2Pg try to order views to avoid error at import time with nested views. With a huge number of views this can take a very long time, you can bypass this ordering by enabling this directive. .IP "\s-1GRANT_OBJECT\s0" 4 .IX Item "GRANT_OBJECT" When exporting GRANTs you can specify a comma separated list of objects for which privilege will be exported. Default is export for all objects. Here are the possibles values \s-1TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY.\s0 Only one object type is allowed at a time. For example set it to \s-1TABLE\s0 if you just want to export privilege on tables. You can use the \-g option to overwrite it. .Sp When used this directive prevent the export of users unless it is set to \s-1USER.\s0 In this case only users definitions are exported. .IP "\s-1WHERE\s0" 4 .IX Item "WHERE" This directive allows you to specify a \s-1WHERE\s0 clause filter when dumping the contents of tables. Value is constructs as follows: TABLE_NAME[\s-1WHERE_CLAUSE\s0], or if you have only one where clause for each table just put the where clause as the value. Both are possible too. Here are some examples: .Sp .Vb 2 \& # Global where clause applying to all tables included in the export \& WHERE 1=1 \& \& # Apply the where clause only on table TABLE_NAME \& WHERE TABLE_NAME[ID1=\*(Aq001\*(Aq] \& \& # Applies two different clause on tables TABLE_NAME and OTHER_TABLE \& # and a generic where clause on DATE_CREATE to all other tables \& WHERE TABLE_NAME[ID1=\*(Aq001\*(Aq OR ID1=\*(Aq002] DATE_CREATE > \*(Aq2001\-01\-01\*(Aq OTHER_TABLE[NAME=\*(Aqtest\*(Aq] .Ve .Sp Any where clause not included into a table name bracket clause will be applied to all exported table including the tables defined in the where clause. These \&\s-1WHERE\s0 clauses are very useful if you want to archive some data or at the opposite only export some recent data. .Sp To be able to quickly test data import it is useful to limit data export to the first thousand tuples of each table. For Oracle define the following clause: .Sp .Vb 1 \& WHERE ROWNUM < 1000 .Ve .Sp and for MySQL, use the following: .Sp .Vb 1 \& WHERE 1=1 LIMIT 1,1000 .Ve .Sp This can also be restricted to some tables data export. .Sp Command line option \-W or \-\-where will override this directive for the global part and per table if the table names is the same. .IP "\s-1TOP_MAX\s0" 4 .IX Item "TOP_MAX" This directive is used to limit the number of item shown in the top N lists like the top list of tables per number of rows and the top list of largest tables in megabytes. By default it is set to 10 items. .IP "\s-1LOG_ON_ERROR\s0" 4 .IX Item "LOG_ON_ERROR" Enable this directive if you want to continue direct data import on error. When Ora2Pg received an error in the \s-1COPY\s0 or \s-1INSERT\s0 statement from PostgreSQL it will log the statement to a file called TABLENAME_error.log in the output directory and continue to next bulk of data. Like this you can try to fix the statement and manually reload the error log file. Default is disabled: abort import on error. .IP "\s-1REPLACE_QUERY\s0" 4 .IX Item "REPLACE_QUERY" Sometime you may want to extract data from an Oracle table but you need a custom query for that. Not just a \*(L"\s-1SELECT\s0 * \s-1FROM\s0 table\*(R" like Ora2Pg do but a more complex query. This directive allows you to overwrite the query used by Ora2Pg to extract data. The format is TABLENAME[\s-1SQL_QUERY\s0]. If you have multiple table to extract by replacing the Ora2Pg query, you can define multiple \s-1REPLACE_QUERY\s0 lines. .Sp .Vb 1 \& REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>\*(Aq2014\-08\-01 00:00:00\*(Aq)] .Ve .SS "Control of Full Text Search export" .IX Subsection "Control of Full Text Search export" Several directives can be used to control the way Ora2Pg will export the Oracle's Text search indexes. By default \s-1CONTEXT\s0 indexes will be exported to PostgreSQL \s-1FTS\s0 indexes but \s-1CTXCAT\s0 indexes will be exported as indexes using the pg_trgm extension. .IP "\s-1CONTEXT_AS_TRGM\s0" 4 .IX Item "CONTEXT_AS_TRGM" Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using pg_trgm extension. Default is to translate \s-1CONTEXT\s0 indexes into \s-1FTS\s0 indexes and \s-1CTXCAT\s0 indexes using pg_trgm. Most of the time using pg_trgm is enough, this is why this directive stand for. You need to create the pg_trgm extension into the destination database before importing the objects: .Sp .Vb 1 \& CREATE EXTENSION pg_trgm; .Ve .IP "\s-1FTS_INDEX_ONLY\s0" 4 .IX Item "FTS_INDEX_ONLY" By default Ora2Pg creates a function-based index to translate Oracle Text indexes. .Sp .Vb 2 \& CREATE INDEX ON t_document \& USING gin(to_tsvector(\*(Aqpg_catalog.french\*(Aq, title)); .Ve .Sp You will have to rewrite the \s-1\fBCONTAIN\s0()\fR clause using \fBto_tsvector()\fR, example: .Sp .Vb 2 \& SELECT id,title FROM t_document \& WHERE to_tsvector(title)) @@ to_tsquery(\*(Aqsearch_word\*(Aq); .Ve .Sp To force Ora2Pg to create an extra tsvector column with a dedicated triggers for \s-1FTS\s0 indexes, disable this directive. In this case, Ora2Pg will add the column as follow: \s-1ALTER TABLE\s0 t_document \s-1ADD COLUMN\s0 tsv_title tsvector; Then update the column to compute \s-1FTS\s0 vectors if data have been loaded before \s-1UPDATE\s0 t_document \s-1SET\s0 tsv_title = to_tsvector('pg_catalog.french', coalesce(title,'')); To automatically update the column when a modification in the title column appears, Ora2Pg adds the following trigger: .Sp .Vb 12 \& CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ \& BEGIN \& IF TG_OP = \*(AqINSERT\*(Aq OR new.title != old.title THEN \& new.tsv_title := \& to_tsvector(\*(Aqpg_catalog.french\*(Aq, coalesce(new.title,\*(Aq\*(Aq)); \& END IF; \& return new; \& END \& $$ LANGUAGE plpgsql; \& CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE \& ON t_document \& FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); .Ve .Sp When the Oracle text index is defined over multiple column, Ora2Pg will use \&\fBsetweight()\fR to set a weight in the order of the column declaration. .IP "\s-1FTS_CONFIG\s0" 4 .IX Item "FTS_CONFIG" Use this directive to force text search configuration to use. When it is not set, Ora2Pg will autodetect the stemmer used by Oracle for each index and pg_catalog.english if the information is not found. .IP "\s-1USE_UNACCENT\s0" 4 .IX Item "USE_UNACCENT" If you want to perform your text search in an accent insensitive way, enable this directive. Ora2Pg will create an helper function over \fBunaccent()\fR and creates the pg_trgm indexes using this function. With \s-1FTS\s0 Ora2Pg will redefine your text search configuration, for example: .Sp .Vb 3 \& CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); \& ALTER TEXT SEARCH CONFIGURATION fr \& ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; .Ve .Sp then set the \s-1FTS_CONFIG\s0 ora2pg.conf directive to fr instead of pg_catalog.english. .Sp When enabled, Ora2pg will create the wrapper function: .Sp .Vb 6 \& CREATE OR REPLACE FUNCTION unaccent_immutable(text) \& RETURNS text AS \& $$ \& SELECT public.unaccent(\*(Aqpublic.unaccent\*(Aq, $1); \& $$ LANGUAGE sql IMMUTABLE \& COST 1; .Ve .Sp the indexes are exported as follow: .Sp .Vb 2 \& CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document \& USING gin (unaccent_immutable(title) gin_trgm_ops); .Ve .Sp In your queries you will need to use the same function in the search to be able to use the function-based index. Example: .Sp .Vb 2 \& SELECT * FROM t_document \& WHERE unaccent_immutable(title) LIKE \*(Aq%donnees%\*(Aq; .Ve .IP "\s-1USE_LOWER_UNACCENT\s0" 4 .IX Item "USE_LOWER_UNACCENT" Same as above but call \fBlower()\fR in the \fBunaccent_immutable()\fR function: .Sp .Vb 5 \& CREATE OR REPLACE FUNCTION unaccent_immutable(text) \& RETURNS text AS \& $$ \& SELECT lower(public.unaccent(\*(Aqpublic.unaccent\*(Aq, $1)); \& $$ LANGUAGE sql IMMUTABLE; .Ve .SS "Modifying object structure" .IX Subsection "Modifying object structure" One of the great usage of Ora2Pg is its flexibility to replicate Oracle database into PostgreSQL database with a different structure or schema. There's three configuration directives that allow you to map those differences. .IP "\s-1REORDERING_COLUMNS\s0" 4 .IX Item "REORDERING_COLUMNS" Enable this directive to reordering columns and minimized the footprint on disc, so that more rows fit on a data page, which is the most important factor for speed. Default is disabled, that mean the same order than in Oracle tables definition, that's should be enough for most usage. This directive is only used with \s-1TABLE\s0 export. .IP "\s-1MODIFY_STRUCT\s0" 4 .IX Item "MODIFY_STRUCT" This directive allows you to limit the columns to extract for a given table. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: .Sp .Vb 1 \& MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... .Ve .Sp for example: .Sp .Vb 1 \& MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) .Ve .Sp This will only extract columns 'id' and 'dossier' from table T_TEST1 and columns \&'id' and 'fichier' from the T_TEST2 table. This directive can only be used with \&\s-1TABLE, COPY\s0 or \s-1INSERT\s0 export. With \s-1TABLE\s0 export create table \s-1DDL\s0 will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. .IP "\s-1EXCLUDE_COLUMNS\s0" 4 .IX Item "EXCLUDE_COLUMNS" Instead of redefining the table structure with \s-1MODIFY_STRUCT\s0 you may want to exclude some columns from the table export. The value consist in a space-separated list of table name with a set of column between parenthesis as follow: .Sp .Vb 1 \& EXCLUDE_COLUMNS NOM_TABLE(nomcol1,nomcol2,...) ... .Ve .Sp for example: .Sp .Vb 1 \& EXCLUDE_COLUMNS T_TEST1(id,dossier) T_TEST2(id,fichier) .Ve .Sp This will exclude from the export columns 'id' and 'dossier' from table T_TEST1 and columns 'id' and 'fichier' from the T_TEST2 table. This directive can only be used with \s-1TABLE, COPY\s0 or \s-1INSERT\s0 export. With \s-1TABLE\s0 export create table \s-1DDL\s0 will respect the new list of columns and all indexes or foreign key pointing to or from a column removed will not be exported. .IP "\s-1REPLACE_TABLES\s0" 4 .IX Item "REPLACE_TABLES" This directive allows you to remap a list of Oracle table name to a PostgreSQL table name during export. The value is a list of space-separated values with the following structure: .Sp .Vb 1 \& REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 .Ve .Sp Oracle tables \s-1ORIG_TBNAME1\s0 and \s-1ORIG_TBNAME2\s0 will be respectively renamed into \&\s-1DEST_TBNAME1\s0 and \s-1DEST_TBNAME2\s0 .IP "\s-1REPLACE_COLS\s0" 4 .IX Item "REPLACE_COLS" Like table name, the name of the column can be remapped to a different name using the following syntax: .Sp .Vb 1 \& REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) .Ve .Sp For example: .Sp .Vb 1 \& REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) .Ve .Sp will rename Oracle columns 'dico' and 'dossier' from table T_TEST into new name \&'dictionary' and 'folder'. .IP "\s-1REPLACE_AS_BOOLEAN\s0" 4 .IX Item "REPLACE_AS_BOOLEAN" If you want to change the type of some Oracle columns into PostgreSQL boolean during the export you can define here a list of tables and column separated by space as follow. .Sp .Vb 1 \& REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 .Ve .Sp The values set in the boolean columns list will be replaced with the 't' and 'f' following the default replacement values and those additionally set in directive \&\s-1BOOLEAN_VALUES.\s0 .Sp Note that if you have modified the table name with \s-1REPLACE_TABLES\s0 and/or the column's name, you need to use the name of the original table and/or column. .Sp .Vb 2 \& REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) \& REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 .Ve .Sp You can also give a type and a precision to automatically convert all fields of that type as a boolean. For example: .Sp .Vb 1 \& REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 .Ve .Sp will also replace any field of type \fBnumber\fR\|(1) or \fBchar\fR\|(1) as a boolean in all exported tables. .IP "\s-1BOOLEAN_VALUES\s0" 4 .IX Item "BOOLEAN_VALUES" Use this to add additional definition of the possible boolean values used in Oracle fields. You must set a space-separated list of \s-1TRUE:FALSE\s0 values. By default here are the values recognized by Ora2Pg: .Sp .Vb 1 \& BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled .Ve .Sp Any values defined here will be added to the default list. .IP "\s-1REPLACE_ZERO_DATE\s0" 4 .IX Item "REPLACE_ZERO_DATE" When Ora2Pg find a \*(L"zero\*(R" date: 0000\-00\-00 00:00:00 it is replaced by a \s-1NULL.\s0 This could be a problem if your column is defined with \s-1NOT NULL\s0 constraint. If you can not remove the constraint, use this directive to set an arbitral date that will be used instead. You can also use \-INFINITY if you don't want to use a fake date. .IP "\s-1INDEXES_SUFFIX\s0" 4 .IX Item "INDEXES_SUFFIX" Add the given value as suffix to indexes names. Useful if you have indexes with same name as tables. For example: .Sp .Vb 1 \& INDEXES_SUFFIX _idx .Ve .Sp will add _idx at ed of all index name. Not so common but can help. .IP "\s-1INDEXES_RENAMING\s0" 4 .IX Item "INDEXES_RENAMING" Enable this directive to rename all indexes using tablename_columns_names. Could be very useful for database that have multiple time the same index name or that use the same name than a table, which is not allowed by PostgreSQL Disabled by default. .IP "\s-1USE_INDEX_OPCLASS\s0" 4 .IX Item "USE_INDEX_OPCLASS" Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B\-tree indexes on the corresponding types. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (\s-1LIKE\s0 or \s-1POSIX\s0 regular expressions) when the database does not use the standard \*(L"C\*(R" locale. If you enable, with value 1, this will force Ora2Pg to export all indexes defined on \fBvarchar2()\fR and \fBchar()\fR columns using those operators. If you set it to a value greater than 1 it will only change indexes on columns where the character limit is greater or equal than this value. For example, set it to 128 to create these kind of indexes on columns of type varchar2(N) where N >= 128. .IP "\s-1RENAME_PARTITION\s0" 4 .IX Item "RENAME_PARTITION" Enable this directive if you want that your partition tables will be renamed. Disabled by default. If you have multiple partitioned table, when exported to PostgreSQL some partitions could have the same name but different parent tables. This is not allowed, table name must be unique, in this case enable this directive. A partition will be renamed following the rule: \*(L"tablename\*(R"_part\*(L"pos\*(R" where \*(L"pos\*(R" is the partition number. For subpartition this is: \*(L"tablename\*(R"_part\*(L"pos\*(R"_subpart\*(L"pos\*(R" If this is partition/subpartition default: \*(L"tablename\*(R"_part_default \*(L"tablename\*(R"_part\*(L"pos\*(R"_subpart_default .IP "\s-1DISABLE_PARTITION\s0" 4 .IX Item "DISABLE_PARTITION" If you don't want to reproduce the partitioning like in Oracle and want to export all partitioned Oracle data into the main single table in PostgreSQL enable this directive. Ora2Pg will export all data into the main table name. Default is to use partitioning, Ora2Pg will export data from each partition and import them into the PostgreSQL dedicated partition table. .IP "\s-1PARTITION_BY_REFERENCE\s0" 4 .IX Item "PARTITION_BY_REFERENCE" How to export partition by reference. Possible values are none, duplicate or the number of hash partition to create. Default is none to not export the partitions by reference. .Sp Value none mean no translation and export of partition by reference like before. Value 'duplicate' will duplicate the referenced column in the partitioned table and apply the same partitioning from the referenced table to the partitioned table. If the value is a number, the table will be partitioned with the \s-1HASH\s0 method using the value as the modulo. For example if you set it to 4 it will create 4 \s-1HASH\s0 partitions. .IP "\s-1DISABLE_UNLOGGED\s0" 4 .IX Item "DISABLE_UNLOGGED" By default Ora2Pg export Oracle tables with the \s-1NOLOGGING\s0 attribute as \&\s-1UNLOGGED\s0 tables. You may want to fully disable this feature because you will lose all data from unlogged tables in case of a PostgreSQL crash. Set it to 1 to export all tables as normal tables. .IP "\s-1DOUBLE_MAX_VARCHAR\s0" 4 .IX Item "DOUBLE_MAX_VARCHAR" Increase varchar max character constraints to support PostgreSQL two bytes character encoding when the source database applies the length constraint on characters not bytes. Default disabled. .SS "Oracle Spatial to PostGis" .IX Subsection "Oracle Spatial to PostGis" Ora2Pg fully export Spatial object from Oracle database. There's some configuration directives that could be used to control the export. .IP "\s-1AUTODETECT_SPATIAL_TYPE\s0" 4 .IX Item "AUTODETECT_SPATIAL_TYPE" By default Ora2Pg is looking at indexes to see the spatial constraint type and dimensions defined under Oracle. Those constraints are passed as at index creation using for example: .Sp .Vb 2 \& CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX \& PARAMETERS(\*(Aqsdo_indx_dims=2, layer_gtype=point\*(Aq); .Ve .Sp If those Oracle constraints parameters are not set, the default is to export those columns as generic type \s-1GEOMETRY\s0 to be able to receive any spatial type. .Sp The \s-1AUTODETECT_SPATIAL_TYPE\s0 directive allows to force Ora2Pg to autodetect the real spatial type and dimension used in a spatial column otherwise a non\- constrained \*(L"geometry\*(R" type is used. Enabling this feature will force Ora2Pg to scan a sample of 50000 column to look at the \s-1GTYPE\s0 used. You can increase or reduce the sample size by setting the value of \s-1AUTODETECT_SPATIAL_TYPE\s0 to the desired number of line to scan. The directive is enabled by default. .Sp For example, in the case of a column named shape and defined with Oracle type \&\s-1SDO_GEOMETRY,\s0 with \s-1AUTODETECT_SPATIAL_TYPE\s0 disabled it will be converted as: .Sp .Vb 1 \& shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) .Ve .Sp and if the directive is enabled and the column just contains a single geometry type that use a single dimension: .Sp .Vb 1 \& shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) .Ve .Sp with a two or three dimensional polygon. .IP "\s-1CONVERT_SRID\s0" 4 .IX Item "CONVERT_SRID" This directive allows you to control the automatically conversion of Oracle \&\s-1SRID\s0 to standard \s-1EPSG.\s0 If enabled, Ora2Pg will use the Oracle function sdo_cs.\fBmap_oracle_srid_to_epsg()\fR to convert all \s-1SRID.\s0 Enabled by default. .Sp If the \s-1SDO_SRID\s0 returned by Oracle is \s-1NULL,\s0 it will be replaced by the default value 8307 converted to its \s-1EPSG\s0 value: 4326 (see \s-1DEFAULT_SRID\s0). .Sp If the value is upper than 1, all \s-1SRID\s0 will be forced to this value, in this case \s-1DEFAULT_SRID\s0 will not be used when Oracle returns a null value and the value will be forced to \s-1CONVERT_SRID.\s0 .Sp Note that it is also possible to set the \s-1EPSG\s0 value on Oracle side when sdo_cs.\fBmap_oracle_srid_to_epsg()\fR return \s-1NULL\s0 if your want to force the value: .Sp .Vb 1 \& system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; .Ve .IP "\s-1DEFAULT_SRID\s0" 4 .IX Item "DEFAULT_SRID" Use this directive to override the default \s-1EPSG SRID\s0 to used: 4326. Can be overwritten by \s-1CONVERT_SRID,\s0 see above. .IP "\s-1GEOMETRY_EXTRACT_TYPE\s0" 4 .IX Item "GEOMETRY_EXTRACT_TYPE" This directive can take three values: \s-1WKT\s0 (default), \s-1WKB\s0 and \s-1INTERNAL.\s0 When it is set to \s-1WKT,\s0 Ora2Pg will use \s-1SDO_UTIL.\fBTO_WKTGEOMETRY\s0()\fR to extract the geometry data. When it is set to \s-1WKB,\s0 Ora2Pg will use the binary output using \s-1SDO_UTIL.\fBTO_WKBGEOMETRY\s0()\fR. If those two extract type are calls at Oracle side, they are slow and you can easily reach Out Of Memory when you have lot of rows. Also \s-1WKB\s0 is not able to export 3D geometry and some geometries like \s-1CURVEPOLYGON.\s0 In this case you may use the \s-1INTERNAL\s0 extraction type. It will use a Pure Perl library to convert the \s-1SDO_GEOMETRY\s0 data into a \s-1WKT\s0 representation, the translation is done on Ora2Pg side. This is a work in progress, please validate your exported data geometries before use. Default spatial object extraction type is \s-1INTERNAL.\s0 .IP "\s-1POSTGIS_SCHEMA\s0" 4 .IX Item "POSTGIS_SCHEMA" Use this directive to add a specific schema to the search path to look for PostGis functions. .IP "\s-1ST_SRID_FUNCTION\s0" 4 .IX Item "ST_SRID_FUNCTION" Oracle function to use to extract the srid from ST_Geometry meta information. Default: \s-1ST_SRID,\s0 for example it should be set to sde.st_srid for ArcSDE. .IP "\s-1ST_DIMENSION_FUNCTION\s0" 4 .IX Item "ST_DIMENSION_FUNCTION" Oracle function to use to extract the dimension from ST_Geometry meta information. Default: \s-1ST_DIMENSION,\s0 for example it should be set to sde.st_dimention for ArcSDE. .IP "\s-1ST_GEOMETRYTYPE_FUNCTION\s0" 4 .IX Item "ST_GEOMETRYTYPE_FUNCTION" Oracle function to use to extract the geometry type from a ST_Geometry column Default: \s-1ST_GEOMETRYTYPE,\s0 for example it should be set to sde.st_geometrytype for ArcSDE. .IP "\s-1ST_ASBINARY_FUNCTION\s0" 4 .IX Item "ST_ASBINARY_FUNCTION" Oracle function to used to convert an ST_Geometry value into \s-1WKB\s0 format. Default: \s-1ST_ASBINARY,\s0 for example it should be set to sde.st_asbinary for ArcSDE. .IP "\s-1ST_ASTEXT_FUNCTION\s0" 4 .IX Item "ST_ASTEXT_FUNCTION" Oracle function to used to convert an ST_Geometry value into \s-1WKT\s0 format. Default: \s-1ST_ASTEXT,\s0 for example it should be set to sde.st_astext for ArcSDE. .SS "PostgreSQL Import" .IX Subsection "PostgreSQL Import" By default conversion to PostgreSQL format is written to file 'output.sql'. The command: .PP .Vb 1 \& psql mydb < output.sql .Ve .PP will import content of file output.sql into PostgreSQL mydb database. .IP "\s-1DATA_LIMIT\s0" 4 .IX Item "DATA_LIMIT" When you are performing \s-1INSERT/COPY\s0 export Ora2Pg proceed by chunks of \s-1DATA_LIMIT\s0 tuples for speed improvement. Tuples are stored in memory before being written to disk, so if you want speed and have enough system resources you can grow this limit to an upper value for example: 100000 or 1000000. Before release 7.0 a value of 0 mean no limit so that all tuples are stored in memory before being flushed to disk. In 7.x branch this has been remove and chunk will be set to the default: 10000 .IP "\s-1BLOB_LIMIT\s0" 4 .IX Item "BLOB_LIMIT" When Ora2Pg detect a table with some \s-1BLOB\s0 it will automatically reduce the value of this directive by dividing it by 10 until his value is below 1000. You can control this value by setting \s-1BLOB_LIMIT.\s0 Exporting \s-1BLOB\s0 use lot of resources, setting it to a too high value can produce \s-1OOM.\s0 .IP "\s-1CLOB_AS_BLOB\s0" 4 .IX Item "CLOB_AS_BLOB" Apply same behavior on \s-1CLOB\s0 than \s-1BLOB\s0 with \s-1BLOB_LIMIT\s0 settings. This is especially useful if you have large \s-1CLOB\s0 data. Default: enabled .IP "\s-1OUTPUT\s0" 4 .IX Item "OUTPUT" The Ora2Pg output filename can be changed with this directive. Default value is output.sql. if you set the file name with extension .gz or .bz2 the output will be automatically compressed. This require that the Compress::Zlib Perl module is installed if the filename extension is .gz and that the bzip2 system command is installed for the .bz2 extension. .IP "\s-1OUTPUT_DIR\s0" 4 .IX Item "OUTPUT_DIR" Since release 7.0, you can define a base directory where the file will be written. The directory must exists. .IP "\s-1BZIP2\s0" 4 .IX Item "BZIP2" This directive allows you to specify the full path to the bzip2 program if it can not be found in the \s-1PATH\s0 environment variable. .IP "\s-1FILE_PER_CONSTRAINT\s0" 4 .IX Item "FILE_PER_CONSTRAINT" Allow object constraints to be saved in a separate file during schema export. The file will be named \s-1CONSTRAINTS_OUTPUT,\s0 where \s-1OUTPUT\s0 is the value of the corresponding configuration directive. You can use .gz xor .bz2 extension to enable compression. Default is to save all data in the \s-1OUTPUT\s0 file. This directive is usable only with \s-1TABLE\s0 export type. .Sp The constraints can be imported quickly into PostgreSQL using the \s-1LOAD\s0 export type to parallelize their creation over multiple (\-j or \s-1JOBS\s0) connections. .IP "\s-1FILE_PER_INDEX\s0" 4 .IX Item "FILE_PER_INDEX" Allow indexes to be saved in a separate file during schema export. The file will be named \s-1INDEXES_OUTPUT,\s0 where \s-1OUTPUT\s0 is the value of the corresponding configuration directive. You can use .gz xor .bz2 file extension to enable compression. Default is to save all data in the \s-1OUTPUT\s0 file. This directive is usable only with \s-1TABLE AND TABLESPACE\s0 export type. With the \s-1TABLESPACE\s0 export, it is used to write \*(L"\s-1ALTER INDEX ... TABLESPACE ...\*(R"\s0 into a separate file named \s-1TBSP_INDEXES_OUTPUT\s0 that can be loaded at end of the migration after the indexes creation to move the indexes. .Sp The indexes can be imported quickly into PostgreSQL using the \s-1LOAD\s0 export type to parallelize their creation over multiple (\-j or \s-1JOBS\s0) connections. .IP "\s-1FILE_PER_FKEYS\s0" 4 .IX Item "FILE_PER_FKEYS" Allow foreign key declaration to be saved in a separate file during schema export. By default foreign keys are exported into the main output file or in the CONSTRAINT_output.sql file. When enabled foreign keys will be exported into a file named FKEYS_output.sql .IP "\s-1FILE_PER_TABLE\s0" 4 .IX Item "FILE_PER_TABLE" Allow data export to be saved in one file per table/view. The files will be named as tablename_OUTPUT, where \s-1OUTPUT\s0 is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the \s-1OUTPUT\s0 directive to enable compression. Default 0 will save all data in one file, set it to 1 to enable this feature. This is usable only during \s-1INSERT\s0 or \s-1COPY\s0 export type. .IP "\s-1FILE_PER_FUNCTION\s0" 4 .IX Item "FILE_PER_FUNCTION" Allow functions, procedures and triggers to be saved in one file per object. The files will be named as objectname_OUTPUT. Where \s-1OUTPUT\s0 is the value of the corresponding configuration directive. You can still use .gz xor .bz2 extension in the \s-1OUTPUT\s0 directive to enable compression. Default 0 will save all in one single file, set it to 1 to enable this feature. This is usable only during the corresponding export type, the package body export has a special behavior. .Sp When export type is \s-1PACKAGE\s0 and you've enabled this directive, Ora2Pg will create a directory per package, named with the lower case name of the package, and will create one file per function/procedure into that directory. If the configuration directive is not enabled, it will create one file per package as packagename_OUTPUT, where \s-1OUTPUT\s0 is the value of the corresponding directive. .IP "\s-1TRUNCATE_TABLE\s0" 4 .IX Item "TRUNCATE_TABLE" If this directive is set to 1, a \s-1TRUNCATE TABLE\s0 instruction will be add before loading data. This is usable only during \s-1INSERT\s0 or \s-1COPY\s0 export type. .Sp When activated, the instruction will be added only if there's no global \s-1DELETE\s0 clause or not one specific to the current table (see below). .IP "\s-1DELETE\s0" 4 .IX Item "DELETE" Support for include a \s-1DELETE FROM ... WHERE\s0 clause filter before importing data and perform a delete of some lines instead of truncating tables. Value is construct as follow: TABLE_NAME[\s-1DELETE_WHERE_CLAUSE\s0], or if you have only one where clause for all tables just put the delete clause as single value. Both are possible too. Here are some examples: .Sp .Vb 3 \& DELETE 1=1 # Apply to all tables and delete all tuples \& DELETE TABLE_TEST[ID1=\*(Aq001\*(Aq] # Apply only on table TABLE_TEST \& DELETE TABLE_TEST[ID1=\*(Aq001\*(Aq OR ID1=\*(Aq002] DATE_CREATE > \*(Aq2001\-01\-01\*(Aq TABLE_INFO[NAME=\*(Aqtest\*(Aq] .Ve .Sp The last applies two different delete where clause on tables \s-1TABLE_TEST\s0 and \&\s-1TABLE_INFO\s0 and a generic delete where clause on \s-1DATE_CREATE\s0 to all other tables. If \s-1TRUNCATE_TABLE\s0 is enabled it will be applied to all tables not covered by the \s-1DELETE\s0 definition. .Sp These \s-1DELETE\s0 clauses might be useful with regular \*(L"updates\*(R". .IP "\s-1STOP_ON_ERROR\s0" 4 .IX Item "STOP_ON_ERROR" Set this parameter to 0 to not include the call to \eset \s-1ON_ERROR_STOP ON\s0 in all \s-1SQL\s0 scripts generated by Ora2Pg. By default this order is always present so that the script will immediately abort when an error is encountered. .IP "\s-1COPY_FREEZE\s0" 4 .IX Item "COPY_FREEZE" Enable this directive to use \s-1COPY FREEZE\s0 instead of a simple \s-1COPY\s0 to export data with rows already frozen. This is intended as a performance option for initial data loading. Rows will be frozen only if the table being loaded has been created or truncated in the current sub-transaction. This will only work with export to file and when \-J or \s-1ORACLE_COPIES\s0 is not set or default to 1. It can be used with direct import into PostgreSQL under the same condition but \-j or \s-1JOBS\s0 must also be unset or default to 1. .IP "\s-1CREATE_OR_REPLACE\s0" 4 .IX Item "CREATE_OR_REPLACE" By default Ora2Pg uses \s-1CREATE OR REPLACE\s0 in functions and views \s-1DDL,\s0 if you need not to override existing functions or views disable this configuration directive, \s-1DDL\s0 will not include \s-1OR REPLACE.\s0 .IP "\s-1DROP_IF_EXISTS\s0" 4 .IX Item "DROP_IF_EXISTS" To add a \s-1DROP\s0 <\s-1OBJECT\s0> \s-1IF EXISTS\s0 before creating the object, enable this directive. Can be useful in an iterative work. Default is disabled. .IP "\s-1EXPORT_GTT\s0" 4 .IX Item "EXPORT_GTT" PostgreSQL do not supports Global Temporary Table natively but you can use the pgtt extension to emulate this behavior. Enable this directive to export global temporary table. .IP "\s-1PGTT_NOSUPERUSER\s0" 4 .IX Item "PGTT_NOSUPERUSER" By default the pgtt extension is loaded using the superuser privilege. Enabled it if you run the \s-1SQL\s0 scripts generated using a non superuser user. It will use: .Sp .Vb 1 \& LOAD \*(Aq$libdir/plugins/pgtt\*(Aq; .Ve .Sp instead of default: .Sp .Vb 1 \& LOAD \*(Aqpgtt\*(Aq; .Ve .IP "\s-1NO_HEADER\s0" 4 .IX Item "NO_HEADER" Enabling this directive will prevent Ora2Pg to print his header into output files. Only the translated code will be written. .IP "\s-1PSQL_RELATIVE_PATH\s0" 4 .IX Item "PSQL_RELATIVE_PATH" By default Ora2Pg use \ei psql command to execute generated \s-1SQL\s0 files if you want to use a relative path following the script execution file enabling this option will use \eir. See psql help for more information. .IP "\s-1DATA_VALIDATION_ROWS\s0" 4 .IX Item "DATA_VALIDATION_ROWS" Number of rows that must be retrieved on both side for data validation. Default it to compare the 10000 first rows. A value of 0 mean compare all rows. .IP "\s-1DATA_VALIDATION_ORDERING\s0" 4 .IX Item "DATA_VALIDATION_ORDERING" Order of rows between both sides are different once the data have been modified. In this case data must be ordered using a primary key or a unique index, that mean that a table without such object can not be compared. If the validation is done just after the data migration without any data modification the validation can be done on all tables without any ordering. .IP "\s-1DATA_VALIDATION_ERROR\s0" 4 .IX Item "DATA_VALIDATION_ERROR" Stop validating data from a table after a certain amount of row mistmatch. Default is to stop after 10 rows validation errors. .IP "\s-1TRANSFORM_VALUE\s0" 4 .IX Item "TRANSFORM_VALUE" Use this directive to precise which transformation should be applied to a column when exporting data. Value must be a semicolon separated list of .Sp .Vb 1 \& TABLE[COLUMN_NAME, ] .Ve .Sp For example to replace string 'Oracle' by 'PostgreSQL' in a varchar2 column use the following. .Sp .Vb 1 \& TRANSFORM_VALUE ERROR_LOG_SAMPLE[DBMS_TYPE:regexp_replace("DBMS_TYPE",\*(AqOracle\*(Aq,\*(AqPostgreSQL\*(Aq)] .Ve .Sp or to replace all Oracle \fBchar\fR\|(0) in a string by a space character: .Sp .Vb 1 \& TRANSFORM_VALUE CLOB_TABLE[CHARDATA:translate("CHARDATA", chr(0), \*(Aq \*(Aq)] .Ve .Sp The expression will be applied in the \s-1SQL\s0 statemeent used to extract data from the source database. .PP When using Ora2Pg export type \s-1INSERT\s0 or \s-1COPY\s0 to dump data to file and that \&\s-1FILE_PER_TABLE\s0 is enabled, you will be warned that Ora2Pg will not export data again if the file already exists. This is to prevent downloading twice table with huge amount of data. To force the download of data from these tables you have to remove the existing output file first. .PP If you want to import data on the fly to the PostgreSQL database you have three configuration directives to set the PostgreSQL database connection. This is only possible with \s-1COPY\s0 or \s-1INSERT\s0 export type as for database schema there's no real interest to do that. .IP "\s-1PG_DSN\s0" 4 .IX Item "PG_DSN" Use this directive to set the PostgreSQL data source namespace using DBD::Pg Perl module as follow: .Sp .Vb 1 \& dbi:Pg:dbname=pgdb;host=localhost;port=5432 .Ve .Sp will connect to database 'pgdb' on localhost at tcp port 5432. .Sp Note that this directive is only used for data export, other export need to be imported manually through the use og psql or any other PostgreSQL client. .Sp To use \s-1SSL\s0 encrypted connection you must add sslmode=require to the connection string like follow: .Sp .Vb 1 \& dbi:Pg:dbname=pgdb;host=localhost;port=5432;sslmode=require .Ve .IP "\s-1PG_USER\s0 and \s-1PG_PWD\s0" 4 .IX Item "PG_USER and PG_PWD" These two directives are used to set the login user and password. .Sp If you do not supply a credential with \s-1PG_PWD\s0 and you have installed the Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If \&\s-1PG_USER\s0 is not set it will be asked interactively too. .IP "\s-1SYNCHRONOUS_COMMIT\s0" 4 .IX Item "SYNCHRONOUS_COMMIT" Specifies whether transaction commit will wait for \s-1WAL\s0 records to be written to disk before the command returns a \*(L"success\*(R" indication to the client. This is the equivalent to set synchronous_commit directive of postgresql.conf file. This is only used when you load data directly to PostgreSQL, the default is off to disable synchronous commit to gain speed at writing data. Some modified version of PostgreSQL, like greenplum, do not have this setting, so in this set this directive to 1, ora2pg will not try to change the setting. .IP "\s-1PG_INITIAL_COMMAND\s0" 4 .IX Item "PG_INITIAL_COMMAND" This directive can be used to send an initial command to PostgreSQL, just after the connection. For example to set some session parameters. This directive can be used multiple times. .IP "\s-1INSERT_ON_CONFLICT\s0" 4 .IX Item "INSERT_ON_CONFLICT" When enabled this instruct Ora2Pg to add an \s-1ON CONFLICT DO NOTHING\s0 clause to all \&\s-1INSERT\s0 statements generated for this type of data export. .SS "Column type control" .IX Subsection "Column type control" .IP "\s-1PG_NUMERIC_TYPE\s0" 4 .IX Item "PG_NUMERIC_TYPE" If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type \s-1NUMBER\s0(p,s) is approximatively converted to real and float PostgreSQL data type. If you have monetary fields or don't want rounding issues with the extra decimals you should preserve the same numeric(p,s) PostgreSQL data type. Do that only if you need exactness because using numeric(p,s) is slower than using real or double. .IP "\s-1PG_INTEGER_TYPE\s0" 4 .IX Item "PG_INTEGER_TYPE" If set to 1 replace portable numeric type into PostgreSQL internal type. Oracle data type \s-1NUMBER\s0(p) or \s-1NUMBER\s0 are converted to smallint, integer or bigint PostgreSQL data type following the value of the precision. If \&\s-1NUMBER\s0 without precision are set to \s-1DEFAULT_NUMERIC\s0 (see below). .IP "\s-1DEFAULT_NUMERIC\s0" 4 .IX Item "DEFAULT_NUMERIC" \&\s-1NUMBER\s0 without precision are converted by default to bigint only if \&\s-1PG_INTEGER_TYPE\s0 is true. You can overwrite this value to any \s-1PG\s0 type, like integer or float. .IP "\s-1DATA_TYPE\s0" 4 .IX Item "DATA_TYPE" If you're experiencing any problem in data type schema conversion with this directive you can take full control of the correspondence between Oracle and PostgreSQL types to redefine data type translation used in Ora2pg. The syntax is a comma-separated list of \*(L"Oracle datatype:Postgresql datatype\*(R". Here are the default list used: .Sp .Vb 1 \& DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,NVARCHAR:varchar,NCHAR:char,DATE:timestamp(0),LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW(16):uuid,RAW(32):uuid,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:integer,INTEGER:integer,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone .Ve .Sp The directive and the list definition must be a single line. .Sp Note that when a \s-1RAW\s0(16) and \s-1RAW\s0(32) columns is found or that the \s-1RAW\s0 column has \*(L"\s-1\fBSYS_GUID\s0()\fR\*(R" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the \*(L"uuid-ossp\*(R" extension. .Sp If you want to replace a type with a precision and scale you need to escape the coma with a backslash. For example, if you want to replace all \s-1NUMBER\s0(*,0) into bigint instead of numeric(38) add the following: .Sp .Vb 1 \& DATA_TYPE NUMBER(*\e,0):bigint .Ve .Sp You don't have to recopy all default type conversion but just the one you want to rewrite. .Sp There's a special case with \s-1BFILE\s0 when they are converted to type \s-1TEXT,\s0 they will just contains the full path to the external file. If you set the destination type to \s-1BYTEA,\s0 the default, Ora2Pg will export the content of the \&\s-1BFILE\s0 as bytea. The third case is when you set the destination type to \s-1EFILE,\s0 in this case, Ora2Pg will export it as an \s-1EFILE\s0 record: (\s-1DIRECTORY, FILENAME\s0). Use the \s-1DIRECTORY\s0 export type to export the existing directories as well as privileges on those directories. .Sp There's no \s-1SQL\s0 function available to retrieve the path to the \s-1BFILE.\s0 Ora2Pg have to create one using the \s-1DBMS_LOB\s0 package. .Sp .Vb 10 \& CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) \& RETURN VARCHAR2 \& AS \& l_dir VARCHAR2(4000); \& l_fname VARCHAR2(4000); \& l_path VARCHAR2(4000); \& BEGIN \& dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); \& SELECT directory_path INTO l_path FROM all_directories \& WHERE directory_name = l_dir; \& l_dir := rtrim(l_path,\*(Aq/\*(Aq); \& RETURN l_dir || \*(Aq/\*(Aq || l_fname; \& END; .Ve .Sp This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and that the destination type is \s-1TEXT.\s0 The function is dropped at the end of the export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. .Sp There's no \s-1SQL\s0 function available to retrieve \s-1BFILE\s0 as an \s-1EFILE\s0 record, then Ora2Pg have to create one using the \s-1DBMS_LOB\s0 package. .Sp .Vb 9 \& CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) \& RETURN VARCHAR2 \& AS \& l_dir VARCHAR2(4000); \& l_fname VARCHAR2(4000); \& BEGIN \& dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); \& RETURN \*(Aq(\*(Aq || l_dir || \*(Aq,\*(Aq || l_fnamei || \*(Aq)\*(Aq; \& END; .Ve .Sp This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and that the destination type is \s-1EFILE.\s0 The function is dropped at the end of the export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. .Sp To set the destination type, use the \s-1DATA_TYPE\s0 configuration directive: .Sp .Vb 1 \& DATA_TYPE BFILE:EFILE .Ve .Sp for example. .Sp The \s-1EFILE\s0 type is a user defined type created by the PostgreSQL extension external_file that can be found here: https://github.com/darold/external_file This is a port of the \s-1BFILE\s0 Oracle type to PostgreSQL. .Sp There's no \s-1SQL\s0 function available to retrieve the content of a \s-1BFILE.\s0 Ora2Pg have to create one using the \s-1DBMS_LOB\s0 package. .Sp .Vb 10 \& CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN \& BLOB \& AS \& filecontent BLOB := NULL; \& src_file BFILE := NULL; \& l_step PLS_INTEGER := 12000; \& l_dir VARCHAR2(4000); \& l_fname VARCHAR2(4000); \& offset NUMBER := 1; \& BEGIN \& IF p_bfile IS NULL THEN \& RETURN NULL; \& END IF; \& \& DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); \& src_file := BFILENAME( l_dir, l_fname ); \& IF src_file IS NULL THEN \& RETURN NULL; \& END IF; \& \& DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); \& DBMS_LOB.CREATETEMPORARY(filecontent, true); \& DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); \& DBMS_LOB.FILECLOSE(src_file); \& RETURN filecontent; \& END; .Ve .Sp This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and that the destination type is bytea (the default). The function is dropped at the end of the export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. .Sp About the \s-1ROWID\s0 and \s-1UROWID,\s0 they are converted into \s-1OID\s0 by \*(L"logical\*(R" default but this will through an error at data import. There is no equivalent data type so you might want to use the \s-1DATA_TYPE\s0 directive to change the corresponding type in PostgreSQL. You should consider replacing this data type by a bigserial (autoincremented sequence), text or uuid data type. .IP "\s-1MODIFY_TYPE\s0" 4 .IX Item "MODIFY_TYPE" Sometimes you need to force the destination type, for example a column exported as timestamp by Ora2Pg can be forced into type date. Value is a comma-separated list of \s-1TABLE:COLUMN:TYPE\s0 structure. If you need to use comma or space inside type definition you will have to backslash them. .Sp .Vb 1 \& MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\e,6) .Ve .Sp Type of table1.col3 will be replaced by a varchar and table1.col4 by a decimal with precision and scale. .Sp If the column's type is a user defined type Ora2Pg will autodetect the composite type and will export its data using \s-1\fBROW\s0()\fR. Some Oracle user defined types are just array of a native type, in this case you may want to transform this column in simple array of a PostgreSQL native type. To do so, just redefine the destination type as wanted and Ora2Pg will also transform the data as an array. For example, with the following definition in Oracle: .Sp .Vb 7 \& CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); \& CREATE TABLE club (Name VARCHAR2(10), \& Address VARCHAR2(20), \& City VARCHAR2(20), \& Phone VARCHAR2(8), \& Members mem_type \& ); .Ve .Sp custom type \*(L"mem_type\*(R" is just a string array and can be translated into the following in PostgreSQL: .Sp .Vb 7 \& CREATE TABLE club ( \& name varchar(10), \& address varchar(20), \& city varchar(20), \& phone varchar(8), \& members text[] \& ) ; .Ve .Sp To do so, just use the directive as follow: .Sp .Vb 1 \& MODIFY_TYPE CLUB:MEMBERS:text[] .Ve .Sp Ora2Pg will take care to transform all data of this column in the correct format. Only arrays of characters and numerics types are supported. .IP "\s-1TO_NUMBER_CONVERSION\s0" 4 .IX Item "TO_NUMBER_CONVERSION" By default Oracle call to function \s-1TO_NUMBER\s0 will be translated as a cast into numeric. For example, \s-1TO_NUMBER\s0('10.1234') is converted into PostgreSQL call to_number('10.1234')::numeric. If you want you can cast the call to integer or bigint by changing the value of the configuration directive. If you need better control of the format, just set it as value, for example: \s-1TO_NUMBER_CONVERSION\s0 99999999999999999999.9999999999 will convert the code above as: \s-1TO_NUMBER\s0('10.1234', '99999999999999999999.9999999999') Any value of the directive that it is not numeric, integer or bigint will be taken as a mask format. If set to none, no conversion will be done. .IP "\s-1VARCHAR_TO_TEXT\s0" 4 .IX Item "VARCHAR_TO_TEXT" By default varchar2 without size constraint are tranlated into text. If you want to keep the varchar name, disable this directive. .IP "\s-1FORCE_IDENTITY_BIGINT\s0" 4 .IX Item "FORCE_IDENTITY_BIGINT" Usually identity column must be bigint to correspond to an auto increment sequence so Ora2Pg always force it to be a bigint. If, for any reason you want Ora2Pg to respect the \s-1DATA_TYPE\s0 you have set for identity column then disable this directive. .IP "\s-1TO_CHAR_NOTIMEZONE\s0" 4 .IX Item "TO_CHAR_NOTIMEZONE" If you want Ora2Pg to remove any timezone information into the format part of the \s-1\fBTO_CHAR\s0()\fR function, enable this directive. Disabled by default. .SS "Taking export under control" .IX Subsection "Taking export under control" The following other configuration directives interact directly with the export process and give you fine granularity in database export control. .IP "\s-1SKIP\s0" 4 .IX Item "SKIP" For \s-1TABLE\s0 export you may not want to export all schema constraints, the \s-1SKIP\s0 configuration directive allows you to specify a space-separated list of constraints that should not be exported. Possible values are: .Sp .Vb 5 \& \- fkeys: turn off foreign key constraints \& \- pkeys: turn off primary keys \& \- ukeys: turn off unique column constraints \& \- indexes: turn off all other index types \& \- checks: turn off check constraints .Ve .Sp For example: .Sp .Vb 1 \& SKIP indexes,checks .Ve .Sp will removed indexes and check constraints from export. .IP "\s-1PKEY_IN_CREATE\s0" 4 .IX Item "PKEY_IN_CREATE" Enable this directive if you want to add primary key definition inside the create table statement. If disabled (the default) primary key definition will be added with an alter table statement. Enable it if you are exporting to GreenPlum PostgreSQL database. .IP "\s-1KEEP_PKEY_NAMES\s0" 4 .IX Item "KEEP_PKEY_NAMES" By default names of the primary and unique key in the source Oracle database are ignored and key names are autogenerated in the target PostgreSQL database with the PostgreSQL internal default naming rules. If you want to preserve Oracle primary and unique key names set this option to 1. .IP "\s-1FKEY_ADD_UPDATE\s0" 4 .IX Item "FKEY_ADD_UPDATE" This directive allows you to add an \s-1ON UPDATE CASCADE\s0 option to a foreign key when a \s-1ON DELETE CASCADE\s0 is defined or always. Oracle do not support this feature, you have to use trigger to operate the \s-1ON UPDATE CASCADE.\s0 As PostgreSQL has this feature, you can choose how to add the foreign key option. There are three values to this directive: never, the default that mean that foreign keys will be declared exactly like in Oracle. The second value is delete, that mean that the \s-1ON UPDATE CASCADE\s0 option will be added only if the \s-1ON DELETE CASCADE\s0 is already defined on the foreign Keys. The last value, always, will force all foreign keys to be defined using the update option. .IP "\s-1FKEY_DEFERRABLE\s0" 4 .IX Item "FKEY_DEFERRABLE" When exporting tables, Ora2Pg normally exports constraints as they are, if they are non-deferrable they are exported as non-deferrable. However, non-deferrable constraints will probably cause problems when attempting to import data to Pg. The \s-1FKEY_DEFERRABLE\s0 option set to 1 will cause all foreign key constraints to be exported as deferrable. .IP "\s-1DEFER_FKEY\s0" 4 .IX Item "DEFER_FKEY" In addition to exporting data when the \s-1DEFER_FKEY\s0 option set to 1, it will add a command to defer all foreign key constraints during data export and the import will be done in a single transaction. This will work only if foreign keys have been exported as deferrable and you are not using direct import to PostgreSQL (\s-1PG_DSN\s0 is not defined). Constraints will then be checked at the end of the transaction. .Sp This directive can also be enabled if you want to force all foreign keys to be created as deferrable and initially deferred during schema export (\s-1TABLE\s0 export type). .IP "\s-1DROP_FKEY\s0" 4 .IX Item "DROP_FKEY" If deferring foreign keys is not possible due to the amount of data in a single transaction, you've not exported foreign keys as deferrable or you are using direct import to PostgreSQL, you can use the \s-1DROP_FKEY\s0 directive. .Sp It will drop all foreign keys before all data import and recreate them at the end of the import. .IP "\s-1DROP_INDEXES\s0" 4 .IX Item "DROP_INDEXES" This directive allows you to gain lot of speed improvement during data import by removing all indexes that are not an automatic index (indexes of primary keys) and recreate them at the end of data import. Of course it is far better to not import indexes and constraints before having imported all data. .IP "\s-1DISABLE_TRIGGERS\s0" 4 .IX Item "DISABLE_TRIGGERS" This directive is used to disable triggers on all tables in \s-1COPY\s0 or \s-1INSERT\s0 export modes. Available values are \s-1USER\s0 (disable user-defined triggers only) and \s-1ALL\s0 (includes \s-1RI\s0 system triggers). Default is 0: do not add \s-1SQL\s0 statements to disable trigger before data import. .Sp If you want to disable triggers during data migration, set the value to \&\s-1USER\s0 if your are connected as non superuser and \s-1ALL\s0 if you are connected as PostgreSQL superuser. A value of 1 is equal to \s-1USER.\s0 .IP "\s-1DISABLE_SEQUENCE\s0" 4 .IX Item "DISABLE_SEQUENCE" If set to 1 it disables alter of sequences on all tables during \s-1COPY\s0 or \s-1INSERT\s0 export mode. This is used to prevent the update of sequence during data migration. Default is 0, alter sequences. .IP "\s-1NOESCAPE\s0" 4 .IX Item "NOESCAPE" By default all data that are not of type date or time are escaped. If you experience any problem with that you can set it to 1 to disable character escaping during data export. This directive is only used during a \s-1COPY\s0 export. See \s-1STANDARD_CONFORMING_STRINGS\s0 for enabling/disabling escape with \s-1INSERT\s0 statements. .IP "\s-1STANDARD_CONFORMING_STRINGS\s0" 4 .IX Item "STANDARD_CONFORMING_STRINGS" This controls whether ordinary string literals ('...') treat backslashes literally, as specified in \s-1SQL\s0 standard. This was the default before Ora2Pg v8.5 so that all strings was escaped first, now this is currently on, causing Ora2Pg to use the escape string syntax (E'...') if this parameter is not set to 0. This is the exact behavior of the same option in PostgreSQL. This directive is only used during data export to build \s-1INSERT\s0 statements. See \s-1NOESCAPE\s0 for enabling/disabling escape in \s-1COPY\s0 statements. .IP "\s-1TRIM_TYPE\s0" 4 .IX Item "TRIM_TYPE" If you want to convert \s-1CHAR\s0(n) from Oracle into varchar(n) or text on PostgreSQL using directive \s-1DATA_TYPE,\s0 you might want to do some trimming on the data. By default Ora2Pg will auto-detect this conversion and remove any whitespace at both leading and trailing position. If you just want to remove the leadings character set the value to \s-1LEADING.\s0 If you just want to remove the trailing character, set the value to \s-1TRAILING.\s0 Default value is \s-1BOTH.\s0 .IP "\s-1TRIM_CHAR\s0" 4 .IX Item "TRIM_CHAR" The default trimming character is space, use this directive if you need to change the character that will be removed. For example, set it to \- if you have leading \- in the char(n) field. To use space as trimming charger, comment this directive, this is the default value. .IP "\s-1PRESERVE_CASE\s0" 4 .IX Item "PRESERVE_CASE" If you want to preserve the case of Oracle object name set this directive to 1. By default Ora2Pg will convert all Oracle object names to lower case. I do not recommend to enable this unless you will always have to double-quote object names on all your \s-1SQL\s0 scripts. .IP "\s-1ORA_RESERVED_WORDS\s0" 4 .IX Item "ORA_RESERVED_WORDS" Allow escaping of column name using Oracle reserved words. Value is a list of comma-separated reserved word. Default: audit,comment,references. .IP "\s-1USE_RESERVED_WORDS\s0" 4 .IX Item "USE_RESERVED_WORDS" Enable this directive if you have table or column names that are a reserved word for PostgreSQL. Ora2Pg will double quote the name of the object. .IP "\s-1GEN_USER_PWD\s0" 4 .IX Item "GEN_USER_PWD" Set this directive to 1 to replace default password by a random password for all extracted user during a \s-1GRANT\s0 export. .IP "\s-1PG_SUPPORTS_MVIEW\s0" 4 .IX Item "PG_SUPPORTS_MVIEW" Since PostgreSQL 9.3, materialized view are supported with the \s-1SQL\s0 syntax \&'\s-1CREATE MATERIALIZED VIEW\s0'. To force Ora2Pg to use the native PostgreSQL support you must enable this configuration \- enable by default. If you want to use the old style with table and a set of function, you should disable it. .IP "\s-1PG_SUPPORTS_IFEXISTS\s0" 4 .IX Item "PG_SUPPORTS_IFEXISTS" PostgreSQL version below 9.x do not support \s-1IF EXISTS\s0 in \s-1DDL\s0 statements. Disabling the directive with value 0 will prevent Ora2Pg to add those keywords in all generated statements. Default value is 1, enabled. .IP "\s-1PG_VERSION\s0" 4 .IX Item "PG_VERSION" Set the PostgreSQL major version number of the target database. Ex: 9.6 or 13 Default is current major version at time of a new release. This replace the old and deprecadted PG_SUPPORTS_* configuration directives described bellow. .IP "\s-1PG_SUPPORTS_ROLE\s0 (Deprecated)" 4 .IX Item "PG_SUPPORTS_ROLE (Deprecated)" This option is deprecated since Ora2Pg release v7.3. .Sp By default Oracle roles are translated into PostgreSQL groups. If you have PostgreSQL 8.1 or more consider the use of \s-1ROLES\s0 and set this directive to 1 to export roles. .IP "\s-1PG_SUPPORTS_INOUT\s0 (Deprecated)" 4 .IX Item "PG_SUPPORTS_INOUT (Deprecated)" This option is deprecated since Ora2Pg release v7.3. .Sp If set to 0, all \s-1IN, OUT\s0 or \s-1INOUT\s0 parameters will not be used into the generated PostgreSQL function declarations (disable it for PostgreSQL database version lower than 8.1), This is now enable by default. .IP "\s-1PG_SUPPORTS_DEFAULT\s0" 4 .IX Item "PG_SUPPORTS_DEFAULT" This directive enable or disable the use of default parameter value in function export. Until PostgreSQL 8.4 such a default value was not supported, this feature is now enable by default. .IP "\s-1PG_SUPPORTS_WHEN\s0 (Deprecated)" 4 .IX Item "PG_SUPPORTS_WHEN (Deprecated)" Add support to \s-1WHEN\s0 clause on triggers as PostgreSQL v9.0 now support it. This directive is enabled by default, set it to 0 disable this feature. .IP "\s-1PG_SUPPORTS_INSTEADOF\s0 (Deprecated)" 4 .IX Item "PG_SUPPORTS_INSTEADOF (Deprecated)" Add support to \s-1INSTEAD OF\s0 usage on triggers (used with \s-1PG\s0 >= 9.1), if this directive is disabled the \s-1INSTEAD OF\s0 triggers will be rewritten as Pg rules. .IP "\s-1PG_SUPPORTS_CHECKOPTION\s0" 4 .IX Item "PG_SUPPORTS_CHECKOPTION" When enabled, export views with \s-1CHECK OPTION.\s0 Disable it if you have PostgreSQL version prior to 9.4. Default: 1, enabled. .IP "\s-1PG_SUPPORTS_IFEXISTS\s0" 4 .IX Item "PG_SUPPORTS_IFEXISTS" If disabled, do not export object with \s-1IF EXISTS\s0 statements. Enabled by default. .IP "\s-1PG_SUPPORTS_PARTITION\s0" 4 .IX Item "PG_SUPPORTS_PARTITION" PostgreSQL version prior to 10.0 do not have native partitioning. Enable this directive if you want to use declarative partitioning. Enable by default. .IP "\s-1PG_SUPPORTS_SUBSTR\s0" 4 .IX Item "PG_SUPPORTS_SUBSTR" Some versions of PostgreSQL like Redshift doesn't support \fBsubstr()\fR and it need to be replaced by a call to \fBsubstring()\fR. In this case, disable it. .IP "\s-1PG_SUPPORTS_NAMED_OPERATOR\s0" 4 .IX Item "PG_SUPPORTS_NAMED_OPERATOR" Disable this directive if you are using \s-1PG\s0 < 9.5, \s-1PL/SQL\s0 operator used in named parameter => will be replaced by PostgreSQL proprietary operator := Enable by default. .IP "\s-1PG_SUPPORTS_IDENTITY\s0" 4 .IX Item "PG_SUPPORTS_IDENTITY" Enable this directive if you have PostgreSQL >= 10 to use \s-1IDENTITY\s0 columns instead of serial or bigserial data type. If \s-1PG_SUPPORTS_IDENTITY\s0 is disabled and there is \s-1IDENTITY\s0 column in the Oracle table, they are exported as serial or bigserial columns. When it is enabled they are exported as \s-1IDENTITY\s0 columns like: .Sp .Vb 4 \& CREATE TABLE identity_test_tab ( \& id bigint GENERATED ALWAYS AS IDENTITY, \& description varchar(30) \& ) ; .Ve .Sp If there is non default sequence options set in Oracle, they will be appended after the \s-1IDENTITY\s0 keyword. Additionally in both cases, Ora2Pg will create a file AUTOINCREMENT_output.sql with a embedded function to update the associated sequences with the restart value set to \*(L"\s-1SELECT\s0 max(colname)+1 \s-1FROM\s0 tablename\*(R". Of course this file must be imported after data import otherwise sequence will be kept to start value. Enabled by default. .IP "\s-1PG_SUPPORTS_PROCEDURE\s0" 4 .IX Item "PG_SUPPORTS_PROCEDURE" PostgreSQL v11 adds support of \s-1PROCEDURE,\s0 enable it if you use such version. .IP "\s-1BITMAP_AS_GIN\s0" 4 .IX Item "BITMAP_AS_GIN" Use btree_gin extension to create bitmap like index with pg >= 9.4 You will need to create the extension by yourself: create extension btree_gin; Default is to create \s-1GIN\s0 index, when disabled, a btree index will be created .IP "\s-1PG_BACKGROUND\s0" 4 .IX Item "PG_BACKGROUND" Use pg_background extension to create an autonomous transaction instead of using a dblink wrapper. With pg >= 9.5 only. Default is to use dblink. See https://github.com/vibhorkum/pg_background about this extension. .IP "\s-1DBLINK_CONN\s0" 4 .IX Item "DBLINK_CONN" By default if you have an autonomous transaction translated using dblink extension instead of pg_background the connection is defined using the values set with \s-1PG_DSN, PG_USER\s0 and \s-1PG_PWD.\s0 If you want to fully override the connection string use this directive as follow to set the connection in the autonomous transaction wrapper function. For example: .Sp .Vb 1 \& DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass .Ve .IP "\s-1LONGREADLEN\s0" 4 .IX Item "LONGREADLEN" Use this directive to set the database handle's 'LongReadLen' attribute to a value that will be the larger than the expected size of the LOBs. The default is 1MB witch may not be enough to extract BLOBs or CLOBs. If the size of the \&\s-1LOB\s0 exceeds the 'LongReadLen' DBD::Oracle will return a '\s-1ORA\-24345: A\s0 Truncation' error. Default: 1023*1024 bytes. .Sp Take a look at this page to learn more: http://search.cpan.org/~pythian/DBD\-Oracle\-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs .Sp Important note: If you increase the value of this directive take care that \&\s-1DATA_LIMIT\s0 will probably needs to be reduced. Even if you only have a 1MB blob, trying to read 10000 of them (the default \s-1DATA_LIMIT\s0) all at once will require 10GB of memory. You may extract data from those table separately and set a \&\s-1DATA_LIMIT\s0 to 500 or lower, otherwise you may experience some out of memory. .IP "\s-1LONGTRUNKOK\s0" 4 .IX Item "LONGTRUNKOK" If you want to bypass the '\s-1ORA\-24345: A\s0 Truncation' error, set this directive to 1, it will truncate the data extracted to the LongReadLen value. Disable by default so that you will be warned if your LongReadLen value is not high enough. .IP "\s-1USE_LOB_LOCATOR\s0" 4 .IX Item "USE_LOB_LOCATOR" Disable this if you want to load full content of \s-1BLOB\s0 and \s-1CLOB\s0 and not use \&\s-1LOB\s0 locators. In this case you will have to set \s-1LONGREADLEN\s0 to the right value. Note that this will not improve speed of \s-1BLOB\s0 export as most of the time is always consumed by the bytea escaping and in this case export is done line by line and not by chunk of \s-1DATA_LIMIT\s0 rows. For more information on how it works, see http://search.cpan.org/~pythian/DBD\-Oracle\-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators .Sp Default is enabled, it use \s-1LOB\s0 locators. .IP "\s-1LOB_CHUNK_SIZE\s0" 4 .IX Item "LOB_CHUNK_SIZE" Oracle recommends reading from and writing to a \s-1LOB\s0 in batches using a multiple of the \s-1LOB\s0 chunk size. This chunk size defaults to 8k (8192). Recent tests shown that the best performances can be reach with higher value like 512K or 4Mb. .Sp A quick benchmark with 30120 rows with different size of \s-1BLOB\s0 (200x5Mb, 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, LONGREADLEN=170Mb and a total table size of 20GB gives: .Sp .Vb 4 \& no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) \& chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) \& chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) \& chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) .Ve .Sp In conclusion it can be more than 10 time faster with \s-1LOB_CHUNK_SIZE\s0 set to 4Mb. Depending of the size of most \s-1BLOB\s0 you may want to adjust the value here. For example if you have a majority of small lobs bellow 8K, using 8192 is better to not waste space. Default value for \s-1LOB_CHUNK_SIZE\s0 is 512000. .IP "\s-1XML_PRETTY\s0" 4 .IX Item "XML_PRETTY" Force the use \fBgetStringVal()\fR instead of \fBgetClobVal()\fR for \s-1XML\s0 data export. Default is 1, enabled for backward compatibility. Set it to 0 to use extract method a la \s-1CLOB.\s0 Note that \s-1XML\s0 value extracted with \fBgetStringVal()\fR must not exceed \s-1VARCHAR2\s0 size limit (4000) otherwise it will return an error. .IP "\s-1ENABLE_MICROSECOND\s0" 4 .IX Item "ENABLE_MICROSECOND" Set it to O if you want to disable export of millisecond from Oracle timestamp columns. By default milliseconds are exported with the use of following format: .Sp .Vb 1 \& \*(AqYYYY\-MM\-DD HH24:MI:SS.FF\*(Aq .Ve .Sp Disabling will force the use of the following Oracle format: .Sp .Vb 1 \& to_char(..., \*(AqYYYY\-MM\-DD HH24:MI:SS\*(Aq) .Ve .Sp By default milliseconds are exported. .IP "\s-1DISABLE_COMMENT\s0" 4 .IX Item "DISABLE_COMMENT" Set this to 1 if you don't want to export comment associated to tables and columns definition. Default is enabled. .SS "Control MySQL export behavior" .IX Subsection "Control MySQL export behavior" .IP "\s-1MYSQL_PIPES_AS_CONCAT\s0" 4 .IX Item "MYSQL_PIPES_AS_CONCAT" Enable this if double pipe and double ampersand (|| and &&) should not be taken as equivalent to \s-1OR\s0 and \s-1AND.\s0 It depend of the variable \f(CW@sql_mode\fR, Use it only if Ora2Pg fail on auto detecting this behavior. .IP "\s-1MYSQL_INTERNAL_EXTRACT_FORMAT\s0" 4 .IX Item "MYSQL_INTERNAL_EXTRACT_FORMAT" Enable this directive if you want \s-1\fBEXTRACT\s0()\fR replacement to use the internal format returned as an integer, for example \s-1DD HH24:MM:SS\s0 will be replaced with format; DDHH24MMSS::bigint, this depend of your apps usage. .SS "Control \s-1SQL\s0 Server export behavior" .IX Subsection "Control SQL Server export behavior" .IP "\s-1DROP_ROWVERSION\s0" 4 .IX Item "DROP_ROWVERSION" PostgreSQL has no equivalent to rowversion datatype and feature, if you want to remove these useless columns, enable this directive. Columns of datatype \&'rowversion' or 'timestamp' will not be exported. .IP "\s-1CASE_INSENSITIVE_SEARCH\s0" 4 .IX Item "CASE_INSENSITIVE_SEARCH" Emulate the same behavior of \s-1MSSQL\s0 with case insensitive search. If the value is citext it will use the citext data type instead of char/varchar/text in tables \s-1DDL\s0 (Ora2Pg will add a \s-1CHECK\s0 constraint for columns with a precision). Instead of citext you can also set a collation name that will be used in the columns definitions. To disable case insensitive search set it to: none. .IP "\s-1SELECT_TOP\s0" 4 .IX Item "SELECT_TOP" Append a \s-1TOP N\s0 clause to the \s-1SELECT\s0 command used to extract the data from \&\s-1SQL\s0 Server. This is the equivalent to a \s-1WHERE ROWNUM\s0 < 1000 clause for Oracle. .SS "Special options to handle character encoding" .IX Subsection "Special options to handle character encoding" .IP "\s-1NLS_LANG\s0 and \s-1NLS_NCHAR\s0" 4 .IX Item "NLS_LANG and NLS_NCHAR" By default Ora2Pg will set \s-1NLS_LANG\s0 to \s-1AMERICAN_AMERICA.AL32UTF8\s0 and \s-1NLS_NCHAR\s0 to \s-1AL32UTF8.\s0 It is not recommended to change those settings but in some case it could be useful. Using your own settings with those configuration directive will change the client encoding at Oracle side by setting the environment variables \&\f(CW$ENV\fR{\s-1NLS_LANG\s0} and \f(CW$ENV\fR{\s-1NLS_NCHAR\s0}. .IP "\s-1BINMODE\s0" 4 .IX Item "BINMODE" By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through a call to the Perl pragma: .Sp .Vb 1 \& use open \*(Aq:utf8\*(Aq; .Ve .Sp You can override this encoding by using the \s-1BINMODE\s0 directive, for example you can set it to :locale to use your locale or iso\-8859\-7, it will respectively use .Sp .Vb 2 \& use open \*(Aq:locale\*(Aq; \& use open \*(Aq:encoding(iso\-8859\-7)\*(Aq; .Ve .Sp If you have change the \s-1NLS_LANG\s0 in non \s-1UTF8\s0 encoding, you might want to set this directive. See http://perldoc.perl.org/5.14.2/open.html for more information. Most of the time, leave this directive commented. .IP "\s-1CLIENT_ENCODING\s0" 4 .IX Item "CLIENT_ENCODING" By default PostgreSQL client encoding is automatically set to \s-1UTF8\s0 to avoid encoding issue. If you have changed the value of \s-1NLS_LANG\s0 you might have to change the encoding of the PostgreSQL client. .Sp You can take a look at the PostgreSQL supported character sets here: http://www.postgresql.org/docs/9.0/static/multibyte.html .IP "\s-1FORCE_PLSQL_ENCODING\s0" 4 .IX Item "FORCE_PLSQL_ENCODING" To force utf8 encoding of the \s-1PL/SQL\s0 code exported, enable this directive. Could be helpful in some rare condition. .SS "\s-1PLSQL\s0 to \s-1PLPGSQL\s0 conversion" .IX Subsection "PLSQL to PLPGSQL conversion" Automatic code conversion from Oracle \s-1PLSQL\s0 to PostgreSQL \s-1PLPGSQL\s0 is a work in progress in Ora2Pg and surely you will always have manual work. The Perl code used for automatic conversion is all stored in a specific Perl Module named Ora2Pg/PLSQL.pm feel free to modify/add you own code and send me patches. The main work in on function, procedure, package and package body headers and parameters rewrite. .IP "\s-1PLSQL_PGSQL\s0" 4 .IX Item "PLSQL_PGSQL" Enable/disable \s-1PLSQL\s0 to \s-1PLPGSQL\s0 conversion. Enabled by default. .IP "\s-1NULL_EQUAL_EMPTY\s0" 4 .IX Item "NULL_EQUAL_EMPTY" Ora2Pg can replace all conditions with a test on \s-1NULL\s0 by a call to the \&\fBcoalesce()\fR function to mimic the Oracle behavior where empty string are considered equal to \s-1NULL.\s0 .Sp .Vb 2 \& (field1 IS NULL) is replaced by (coalesce(field1::text, \*(Aq\*(Aq) = \*(Aq\*(Aq) \& (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> \*(Aq\*(Aq) .Ve .Sp You might want this replacement to be sure that your application will have the same behavior but if you have control on you application a better way is to change it to transform empty string into \s-1NULL\s0 because PostgreSQL makes the difference. .IP "\s-1EMPTY_LOB_NULL\s0" 4 .IX Item "EMPTY_LOB_NULL" Force \fBempty_clob()\fR and \fBempty_blob()\fR to be exported as \s-1NULL\s0 instead as empty string for the first one and '\ex' for the second. If \s-1NULL\s0 is allowed in your column this might improve data export speed if you have lot of empty lob. Default is to preserve the exact data from Oracle. .IP "\s-1PACKAGE_AS_SCHEMA\s0" 4 .IX Item "PACKAGE_AS_SCHEMA" If you don't want to export package as schema but as simple functions you might also want to replace all call to package_name.function_name. If you disable the \s-1PACKAGE_AS_SCHEMA\s0 directive then Ora2Pg will replace all call to package_name.\fBfunction_name()\fR by \fBpackage_name_function_name()\fR. Default is to use a schema to emulate package. .Sp The replacement will be done in all kind of \s-1DDL\s0 or code that is parsed by the \s-1PLSQL\s0 to \s-1PLPGSQL\s0 converter. \s-1PLSQL_PGSQL\s0 must be enabled or \-p used in command line. .IP "\s-1REWRITE_OUTER_JOIN\s0" 4 .IX Item "REWRITE_OUTER_JOIN" Enable this directive if the rewrite of Oracle native syntax (+) of \&\s-1OUTER JOIN\s0 is broken. This will force Ora2Pg to not rewrite such code, default is to try to rewrite simple form of right outer join for the moment. .IP "\s-1UUID_FUNCTION\s0" 4 .IX Item "UUID_FUNCTION" By default Ora2Pg will convert call to \s-1\fBSYS_GUID\s0()\fR Oracle function with a call to uuid_generate_v4 from uuid-ossp extension. You can redefined it to use the gen_random_uuid function from pgcrypto extension by changing the function name. Default to uuid_generate_v4. .Sp Note that when a \s-1RAW\s0(16) and \s-1RAW\s0(32) columns is found or that the \s-1RAW\s0 column has \*(L"\s-1\fBSYS_GUID\s0()\fR\*(R" as default value Ora2Pg will automatically translate the type of the column into uuid which might be the right translation in most of the case. In this case data will be automatically migrated as PostgreSQL uuid data type provided by the \*(L"uuid-ossp\*(R" extension. .IP "\s-1FUNCTION_STABLE\s0" 4 .IX Item "FUNCTION_STABLE" By default Oracle functions are marked as \s-1STABLE\s0 as they can not modify data unless when used in \s-1PL/SQL\s0 with variable assignment or as conditional expression. You can force Ora2Pg to create these function as \s-1VOLATILE\s0 by disabling this configuration directive. .IP "\s-1COMMENT_COMMIT_ROLLBACK\s0" 4 .IX Item "COMMENT_COMMIT_ROLLBACK" By default call to \s-1COMMIT/ROLLBACK\s0 are kept untouched by Ora2Pg to force the user to review the logic of the function. Once it is fixed in Oracle source code or you want to comment this calls enable the following directive. .IP "\s-1COMMENT_SAVEPOINT\s0" 4 .IX Item "COMMENT_SAVEPOINT" It is common to see \s-1SAVEPOINT\s0 call inside \s-1PL/SQL\s0 procedure together with a \s-1ROLLBACK TO\s0 savepoint_name. When \s-1COMMENT_COMMIT_ROLLBACK\s0 is enabled you may want to also comment \s-1SAVEPOINT\s0 calls, in this case enable it. .IP "\s-1STRING_CONSTANT_REGEXP\s0" 4 .IX Item "STRING_CONSTANT_REGEXP" Ora2Pg replace all string constant during the pl/sql to plpgsql translation, string constant are all text include between single quote. If you have some string placeholder used in dynamic call to queries you can set a list of regexp to be temporary replaced to not break the parser. For example: .Sp .Vb 1 \& STRING_CONSTANT_REGEXP .Ve .Sp The list of regexp must use the semi colon as separator. .IP "\s-1ALTERNATIVE_QUOTING_REGEXP\s0" 4 .IX Item "ALTERNATIVE_QUOTING_REGEXP" To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals set the regexp with the text capture to use to extract the text part. For example with a variable declared as .Sp .Vb 1 \& c_sample VARCHAR2(100 CHAR) := q\*(Aq{This doesn\*(Aqt work.}\*(Aq; .Ve .Sp the regexp to use must be: .Sp .Vb 1 \& ALTERNATIVE_QUOTING_REGEXP q\*(Aq{(.*)}\*(Aq .Ve .Sp ora2pg will use the $$ delimiter, with the example the result will be: .Sp .Vb 1 \& c_sample varchar(100) := $$This doesn\*(Aqt work.$$; .Ve .Sp The value of this configuration directive can be a list of regexp separated by a semi colon. The capture part (between parenthesis) is mandatory in each regexp if you want to restore the string constant. .IP "\s-1USE_ORAFCE\s0" 4 .IX Item "USE_ORAFCE" If you want to use functions defined in the Orafce library and prevent Ora2Pg to translate call to these functions, enable this directive. The Orafce library can be found here: https://github.com/orafce/orafce .Sp By default Ora2pg rewrite \fBadd_month()\fR, \fBadd_year()\fR, \fBdate_trunc()\fR and \&\fBto_char()\fR functions, but you may prefer to use the orafce version of these function that do not need any code transformation. .IP "\s-1AUTONOMOUS_TRANSACTION\s0" 4 .IX Item "AUTONOMOUS_TRANSACTION" Enable translation of autonomous transactions into a wrapper function using dblink or pg_background extension. If you don't want to use this translation and just want the function to be exported as a normal one without the pragma call, disable this directive. .SS "Materialized view" .IX Subsection "Materialized view" Materialized views are exported as snapshot \*(L"Snapshot Materialized Views\*(R" as PostgreSQL only supports full refresh. .PP If you want to import the materialized views in PostgreSQL prior to 9.3 you have to set configuration directive \s-1PG_SUPPORTS_MVIEW\s0 to 0. In this case Ora2Pg will export all materialized views as explain in this document: .PP .Vb 1 \& http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. .Ve .PP When exporting materialized view Ora2Pg will first add the \s-1SQL\s0 code to create the \*(L"materialized_views\*(R" table: .PP .Vb 6 \& CREATE TABLE materialized_views ( \& mview_name text NOT NULL PRIMARY KEY, \& view_name text NOT NULL, \& iname text, \& last_refresh TIMESTAMP WITH TIME ZONE \& ); .Ve .PP all materialized views will have an entry in this table. It then adds the plpgsql code to create tree functions: .PP .Vb 3 \& create_materialized_view(text, text, text) used to create a materialized view \& drop_materialized_view(text) used to delete a materialized view \& refresh_full_materialized_view(text) used to refresh a view .Ve .PP then it adds the \s-1SQL\s0 code to create the view and the materialized view: .PP .Vb 2 \& CREATE VIEW mviewname_mview AS \& SELECT ... FROM ...; \& \& SELECT create_materialized_view(\*(Aqmviewname\*(Aq,\*(Aqmviewname_mview\*(Aq, change with the name of the column to used for the index); .Ve .PP The first argument is the name of the materialized view, the second the name of the view on which the materialized view is based and the third is the column name on which the index should be build (aka most of the time the primary key). This column is not automatically deduced so you need to replace its name. .PP As said above Ora2Pg only supports snapshot materialized views so the table will be entirely refreshed by issuing first a truncate of the table and then by load again all data from the view: .PP .Vb 1 \& refresh_full_materialized_view(\*(Aqmviewname\*(Aq); .Ve .PP To drop the materialized view you just have to call the \fBdrop_materialized_view()\fR function with the name of the materialized view as parameter. .SS "Other configuration directives" .IX Subsection "Other configuration directives" .IP "\s-1DEBUG\s0" 4 .IX Item "DEBUG" Set it to 1 will enable verbose output. .IP "\s-1IMPORT\s0" 4 .IX Item "IMPORT" You can define common Ora2Pg configuration directives into a single file that can be imported into other configuration files with the \s-1IMPORT\s0 configuration directive as follow: .Sp .Vb 1 \& IMPORT commonfile.conf .Ve .Sp will import all configuration directives defined into commonfile.conf into the current configuration file. .SS "Exporting views as PostgreSQL tables" .IX Subsection "Exporting views as PostgreSQL tables" You can export any Oracle view as a PostgreSQL table simply by setting \s-1TYPE\s0 configuration option to \s-1TABLE\s0 to have the corresponding create table statement. Or use type \s-1COPY\s0 or \s-1INSERT\s0 to export the corresponding data. To allow that you have to specify your views in the \s-1VIEW_AS_TABLE\s0 configuration option. .PP Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) into a \s-1PG\s0 create table form, then it will extract the data (if TYPE=COPY or \s-1INSERT\s0) following the view schema. .PP For example, with the following view: .PP .Vb 6 \& CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS \& SELECT category_id, COUNT(*) as product_count, \& MIN(list_price) as low_price, \& MAX(list_price) as high_price \& FROM product_information \& GROUP BY category_id; .Ve .PP Setting \s-1VIEW_AS_TABLE\s0 to product_prices and using export type \s-1TABLE,\s0 will force Ora2Pg to detect columns returned types and to generate a create table statement: .PP .Vb 6 \& CREATE TABLE product_prices ( \& category_id bigint, \& product_count integer, \& low_price numeric, \& high_price numeric \& ); .Ve .PP Data will be loaded following the \s-1COPY\s0 or \s-1INSERT\s0 export type and the view declaration. .PP You can use the \s-1ALLOW\s0 and \s-1EXCLUDE\s0 directive in addition to filter other objects to export. .SS "Export as Kettle transformation \s-1XML\s0 files" .IX Subsection "Export as Kettle transformation XML files" The \s-1KETTLE\s0 export type is useful if you want to use Penthalo Data Integrator (Kettle) to import data to PostgreSQL. With this type of export Ora2Pg will generate one \s-1XML\s0 Kettle transformation files (.ktr) per table and add a line to manually execute the transformation in the output.sql file. For example: .PP .Vb 1 \& ora2pg \-c ora2pg.conf \-t KETTLE \-j 12 \-a MYTABLE \-o load_mydata.sh .Ve .PP will generate one file called '\s-1HR.MYTABLE\s0.ktr' and add a line to the output file (load_mydata.sh): .PP .Vb 1 \& #!/bin/sh \& \& KETTLE_TEMPLATE_PATH=\*(Aq.\*(Aq \& \& JAVAMAXMEM=4096 ./pan.sh \-file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr \-level Detailed .Ve .PP The \-j 12 option will create a template with 12 processes to insert data into PostgreSQL. It is also possible to specify the number of parallel queries used to extract data from the Oracle with the \-J command line option as follow: .PP .Vb 1 \& ora2pg \-c ora2pg.conf \-t KETTLE \-J 4 \-j 12 \-a EMPLOYEES \-o load_mydata.sh .Ve .PP This is only possible if there is a unique key defined on a numeric column or that you have defined the technical key to used to split the query between cores in the \s-1DEFINED_PKEY\s0 configuration directive. For example: .PP .Vb 1 \& DEFINED_PK EMPLOYEES:employee_id .Ve .PP will force the number of Oracle connection copies to 4 and defined the \s-1SQL\s0 query as follow in the Kettle \s-1XML\s0 transformation file: .PP .Vb 1 \& SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} .Ve .PP The \s-1KETTLE\s0 export type requires that the Oracle and PostgreSQL \s-1DSN\s0 are defined. You can also activate the \s-1TRUNCATE_TABLE\s0 directive to force a truncation of the table before data import. .PP The \s-1KETTLE\s0 export type is an original work of Marc Cousin. .SS "Migration cost assessment" .IX Subsection "Migration cost assessment" Estimating the cost of a migration process from Oracle to PostgreSQL is not easy. To obtain a good assessment of this migration cost, Ora2Pg will inspect all database objects, all functions and stored procedures to detect if there's still some objects and \s-1PL/SQL\s0 code that can not be automatically converted by Ora2Pg. .PP Ora2Pg has a content analysis mode that inspect the Oracle database to generate a text report on what the Oracle database contains and what can not be exported. .PP To activate the \*(L"analysis and report\*(R" mode, you have to use the export de type \&\s-1SHOW_REPORT\s0 like in the following command: .PP .Vb 1 \& ora2pg \-t SHOW_REPORT .Ve .PP Here is a sample report obtained with this command: .PP .Vb 6 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Ora2Pg: Oracle Database Content Report \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 \& Schema HR \& Size 880.00 MB \& \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Object Number Invalid Comments \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& CLUSTER 2 0 Clusters are not supported and will not be exported. \& FUNCTION 40 0 Total size of function code: 81992. \& INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will \& do so on PostgreSQL. 1 bitmap index(es). 230 b\-tree index(es). 1 reversed b\-tree index(es) \& Note that bitmap index(es) will be exported as b\-tree index(es) if any. Cluster, domain, \& bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported \& too, you may use a trigram\-based index (see pg_trgm) or a reverse() function based index \& and search. You may also use \*(Aqvarchar_pattern_ops\*(Aq, \*(Aqtext_pattern_ops\*(Aq or \*(Aqbpchar_pattern_ops\*(Aq \& operators in your indexes to improve search with the LIKE operator respectively into \& varchar, text or char columns. \& MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they \& are only updated when fully refreshed. \& PACKAGE BODY 2 1 Total size of package code: 20700. \& PROCEDURE 7 0 Total size of procedure code: 19198. \& SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL \& will be transformed into NEXTVAL(\*(Aqsequence_name\*(Aq) or CURRVAL(\*(Aqsequence_name\*(Aq). \& TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration \& directive to export as file_fdw foreign tables or use COPY in your code if you just \& want to load data from external files. 2 binary columns. 4 unknown types. \& TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. \& 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. \& TRIGGER 30 0 Total size of trigger code: 21677. \& TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. \& 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type \& inherited and Subtype are converted as table, type inheritance is not supported. \& TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. \& VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use \& INSTEAD OF triggers. \& DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use \& the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. \& \& Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. .Ve .PP Once the database can be analysed, Ora2Pg, by his ability to convert \s-1SQL\s0 and \s-1PL/SQL\s0 code from Oracle syntax to PostgreSQL, can go further by estimating the code difficulties and estimate the time necessary to operate a full database migration. .PP To estimate the migration cost in person-days, Ora2Pg allow you to use a configuration directive called \s-1ESTIMATE_COST\s0 that you can also enabled at command line: .PP .Vb 1 \& \-\-estimate_cost .Ve .PP This feature can only be used with the \s-1SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE\s0 and \s-1QUERY\s0 export type. .PP .Vb 1 \& ora2pg \-t SHOW_REPORT \-\-estimate_cost .Ve .PP The generated report is same as above but with a new 'Estimated cost' column as follow: .PP .Vb 6 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Ora2Pg: Oracle Database Content Report \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Version Oracle Database 10g Express Edition Release 10.2.0.1.0 \& Schema HR \& Size 890.00 MB \& \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Object Number Invalid Estimated cost Comments \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL\*(Aqs Foreign Data Wrapper (FDW) extensions \& using oracle_fdw. \& FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. \& INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so \& on PostgreSQL. 11 b\-tree index(es). Note that bitmap index(es) will be exported as b\-tree \& index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. \& Reverse indexes are not exported too, you may use a trigram\-based index (see pg_trgm) or a \& reverse() function based index and search. You may also use \*(Aqvarchar_pattern_ops\*(Aq, \*(Aqtext_pattern_ops\*(Aq \& or \*(Aqbpchar_pattern_ops\*(Aq operators in your indexes to improve search with the LIKE operator \& respectively into varchar, text or char columns. \& JOB 0 0 0 Job are not exported. You may set external cron job with them. \& MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they \& are only updated when fully refreshed. \& PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found \& inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, \& emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, \& emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. \& PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, \& PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. \& SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL \& will be transformed into NEXTVAL(\*(Aqsequence_name\*(Aq) or CURRVAL(\*(Aqsequence_name\*(Aq). \& SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround \& is to use views or set the PostgreSQL search_path in your session to access \& object outside the current schema. \& user1.emp_details_view_v is an alias to hr.emp_details_view. \& user1.emp_table is an alias to hr.employees@other_server. \& user1.offices is an alias to hr.locations. \& TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration \& directive to export as file_fdw foreign tables or use COPY in your code if you just want to \& load data from external files. 2 binary columns. 4 unknown types. \& TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. \& TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. \& 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are \& converted as table, type inheritance is not supported. \& TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. \& VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). .Ve .PP The last line shows the total estimated migration cost in person-days following the number of migration units estimated for each object. This migration unit represent around five minutes for a PostgreSQL expert. If this is your first migration you can get it higher with the configuration directive \s-1COST_UNIT_VALUE\s0 or the \-\-cost_unit_value command line option: .PP .Vb 1 \& ora2pg \-t SHOW_REPORT \-\-estimate_cost \-\-cost_unit_value 10 .Ve .PP Ora2Pg is also able to give you a migration difficulty level assessment, here a sample: .PP Migration level: B\-5 .PP .Vb 10 \& Migration levels: \& A \- Migration that might be run automatically \& B \- Migration with code rewrite and a person\-days cost up to 5 days \& C \- Migration with code rewrite and a person\-days cost above 5 days \& Technical levels: \& 1 = trivial: no stored functions and no triggers \& 2 = easy: no stored functions but with triggers, no manual rewriting \& 3 = simple: stored functions and/or triggers, no manual rewriting \& 4 = manual: no stored functions but with triggers or views with code rewriting \& 5 = difficult: stored functions and/or triggers with code rewriting .Ve .PP This assessment consist in a letter A or B to specify if the migration needs manual rewriting or not. And a number from 1 up to 5 to give you a technical difficulty level. You have an additional option \-\-human_days_limit to specify the number of person-days limit where the migration level should be set to C to indicate that it need a huge amount of work and a full project management with migration support. Default is 10 person-days. You can use the configuration directive \s-1HUMAN_DAYS_LIMIT\s0 to change this default value permanently. .PP This feature has been developed to help you or your boss to decide which database to migrate first and the team that must be mobilized to operate the migration. .SS "Global Oracle and MySQL migration assessment" .IX Subsection "Global Oracle and MySQL migration assessment" Ora2Pg come with a script ora2pg_scanner that can be used when you have a huge number of instances and schema to scan for migration assessment. .PP Usage: ora2pg_scanner \-l \s-1CSVFILE\s0 [\-o \s-1OUTDIR\s0] .PP .Vb 8 \& \-b | \-\-binpath DIR: full path to directory where the ora2pg binary stays. \& Might be useful only on Windows OS. \& \-c | \-\-config FILE: set custom configuration file to use otherwise ora2pg \& will use the default: /etc/ora2pg/ora2pg.conf. \& \-l | \-\-list FILE : CSV file containing a list of databases to scan with \& all required information. The first line of the file \& can contain the following header that describes the \& format that must be used: \& \& "type","schema/database","dsn","user","password" \& \& \-o | \-\-outdir DIR : (optional) by default all reports will be dumped to a \& directory named \*(Aqoutput\*(Aq, it will be created automatically. \& If you want to change the name of this directory, set the name \& at second argument. \& \& \-t | \-\-test : just try all connections by retrieving the required schema \& or database name. Useful to validate your CSV list file. \& \-u | \-\-unit MIN : redefine globally the migration cost unit value in minutes. \& Default is taken from the ora2pg.conf (default 5 minutes). \& \& Here is a full example of a CSV databases list file: \& \& "type","schema/database","dsn","user","password" \& "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" \& "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" \& "MSSQL","HR","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","system","manager" \& \& The CSV field separator must be a comma. \& \& Note that if you want to scan all schemas from an Oracle instance you just \& have to leave the schema field empty, Ora2Pg will automatically detect all \& available schemas and generate a report for each one. Of course you need to \& use a connection user with enough privileges to be able to scan all schemas. \& For example: \& \& "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" \& "MSSQL","","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","usrname","passwd" \& \& will generate a report for all schema in the XE instance. Note that in this \& case the SCHEMA directive in ora2pg.conf must not be set. .Ve .PP It will generate a \s-1CSV\s0 file with the assessment result, one line per schema or database and a detailed \s-1HTML\s0 report for each database scanned. .PP Hint: Use the \-t | \-\-test option before to test all your connections in your \&\s-1CSV\s0 file. .PP For Windows users you must use the \-b command line option to set the directory where ora2pg_scanner stays otherwise the ora2pg command calls will fail. .PP In the migration assessment details about functions Ora2Pg always include per default 2 migration units for \s-1TEST\s0 and 1 unit for \s-1SIZE\s0 per 1000 characters in the code. This mean that by default it will add 15 minutes in the migration assessment per function. Obviously if you have unitary tests or very simple functions this will not represent the real migration time. .SS "Migration assessment method" .IX Subsection "Migration assessment method" Migration unit scores given to each type of Oracle database object are defined in the Perl library lib/Ora2Pg/PLSQL.pm in the \f(CW%OBJECT_SCORE\fR variable definition. .PP The number of \s-1PL/SQL\s0 lines associated to a migration unit is also defined in this file in the \f(CW$SIZE_SCORE\fR variable value. .PP The number of migration units associated to each \s-1PL/SQL\s0 code difficulties can be found in the same Perl library lib/Ora2Pg/PLSQL.pm in the hash \f(CW%UNCOVERED_SCORE\fR initialization. .PP This assessment method is a work in progress so I'm expecting feedbacks on migration experiences to polish the scores/units attributed in those variables. .SS "Improving indexes and constraints creation speed" .IX Subsection "Improving indexes and constraints creation speed" Using the \s-1LOAD\s0 export type and a file containing \s-1SQL\s0 orders to perform, it is possible to dispatch those orders over multiple PostgreSQL connections. To be able to use this feature, the \s-1PG_DSN, PG_USER\s0 and \s-1PG_PWD\s0 must be set. Then: .PP .Vb 1 \& ora2pg \-t LOAD \-c config/ora2pg.conf \-i schema/tables/INDEXES_table.sql \-j 4 .Ve .PP will dispatch indexes creation over 4 simultaneous PostgreSQL connections. .PP This will considerably accelerate this part of the migration process with huge data size. .SS "Exporting \s-1LONG RAW\s0" .IX Subsection "Exporting LONG RAW" If you still have columns defined as \s-1LONG RAW,\s0 Ora2Pg will not be able to export these kind of data. The \s-1OCI\s0 library fail to export them and always return the same first record. To be able to export the data you need to transform the field as \s-1BLOB\s0 by creating a temporary table before migrating data. For example, the Oracle table: .PP .Vb 5 \& SQL> DESC TEST_LONGRAW \& Name NULL ? Type \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \& ID NUMBER \& C1 LONG RAW .Ve .PP need to be \*(L"translated\*(R" into a table using \s-1BLOB\s0 as follow: .PP .Vb 1 \& CREATE TABLE test_blob (id NUMBER, c1 BLOB); .Ve .PP And then copy the data with the following \s-1INSERT\s0 query: .PP .Vb 1 \& INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; .Ve .PP Then you just have to exclude the original table from the export (see \s-1EXCLUDE\s0 directive) and to renamed the new temporary table on the fly using the \&\s-1REPLACE_TABLES\s0 configuration directive. .SS "Global variables" .IX Subsection "Global variables" Oracle allow the use of global variables defined in packages. Ora2Pg will export these variables for PostgreSQL as user defined custom variables available in a session. Oracle variables assignment are exported as call to: .PP .Vb 1 \& PERFORM set_config(\*(Aqpkgname.varname\*(Aq, value, false); .Ve .PP Use of these variables in the code is replaced by: .PP .Vb 1 \& current_setting(\*(Aqpkgname.varname\*(Aq)::global_variables_type; .Ve .PP where global_variables_type is the type of the variable extracted from the package definition. .PP If the variable is a constant or have a default value assigned at declaration, Ora2Pg will create a file global_variables.conf with the definition to include in the postgresql.conf file so that their values will already be set at database connection. Note that the value can always modified by the user so you can not have exactly a constant. .SS "Hints" .IX Subsection "Hints" Converting your queries with Oracle style outer join (+) syntax to \s-1ANSI\s0 standard \s-1SQL\s0 at the Oracle side can save you lot of time for the migration. You can use \s-1TOAD\s0 Query Builder can re-write these using the proper \s-1ANSI\s0 syntax, see: http://www.toadworld.com/products/toad\-for\-oracle/f/10/t/9518.aspx .PP There's also an alternative with \s-1SQL\s0 Developer Data Modeler, see http://www.thatjeffsmith.com/archive/2012/01/sql\-developer\-data\-modeler\-quick\-tip\-use\-oracle\-join\-syntax\-or\-ansi/ .PP Toad is also able to rewrite the native Oracle \s-1\fBDECODE\s0()\fR syntax into \s-1ANSI\s0 standard \s-1SQL CASE\s0 statement. You can find some slide about this in a presentation given at PgConf.RU: http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf .SS "Test the migration" .IX Subsection "Test the migration" The type of action called \s-1TEST\s0 allow you to check that all objects from Oracle database have been created under PostgreSQL. Of course \s-1PG_DSN\s0 must be set to be able to check PostgreSQL side. .PP Note that this feature respect the schema name limitation if \s-1EXPORT_SCHEMA\s0 and \&\s-1SCHEMA\s0 or \s-1PG_SCHEMA\s0 are defined. If only \s-1EXPORT_SCHEMA\s0 is set all schemes from Oracle and PostgreSQL are scanned. You can filter to a single schema using \&\s-1SCHEMA\s0 and/or \s-1PG_SCHEMA\s0 but you can not filter on a list of schema. To test a list of schema you will have to repeat the calls to Ora2Pg by specifying a single schema each time. .PP For example command: .PP .Vb 1 \& ora2pg \-t TEST \-c config/ora2pg.conf > migration_diff.txt .Ve .PP Will create a file containing the report of all object and row count on both side, Oracle and PostgreSQL, with an error section giving you the detail of the differences for each kind of object. Here is a sample result: .PP .Vb 7 \& [TEST INDEXES COUNT] \& ORACLEDB:DEPARTMENTS:2 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:6 \& POSTGRES:employees:6 \& [ERRORS INDEXES COUNT] \& Table departments doesn\*(Aqt have the same number of indexes in Oracle (2) and in PostgreSQL (1). \& \& [TEST UNIQUE CONSTRAINTS COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS UNIQUE CONSTRAINTS COUNT] \& OK, Oracle and PostgreSQL have the same number of unique constraints. \& \& [TEST PRIMARY KEYS COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS PRIMARY KEYS COUNT] \& OK, Oracle and PostgreSQL have the same number of primary keys. \& \& [TEST CHECK CONSTRAINTS COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS CHECK CONSTRAINTS COUNT] \& OK, Oracle and PostgreSQL have the same number of check constraints. \& \& [TEST NOT NULL CONSTRAINTS COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS NOT NULL CONSTRAINTS COUNT] \& OK, Oracle and PostgreSQL have the same number of not null constraints. \& \& [TEST COLUMN DEFAULT VALUE COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS COLUMN DEFAULT VALUE COUNT] \& OK, Oracle and PostgreSQL have the same number of column default value. \& \& [TEST IDENTITY COLUMN COUNT] \& ORACLEDB:DEPARTMENTS:1 \& POSTGRES:departments:1 \& ORACLEDB:EMPLOYEES:0 \& POSTGRES:employees:0 \& [ERRORS IDENTITY COLUMN COUNT] \& OK, Oracle and PostgreSQL have the same number of identity column. \& \& [TEST FOREIGN KEYS COUNT] \& ORACLEDB:DEPARTMENTS:0 \& POSTGRES:departments:0 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS FOREIGN KEYS COUNT] \& OK, Oracle and PostgreSQL have the same number of foreign keys. \& \& [TEST TABLE COUNT] \& ORACLEDB:TABLE:2 \& POSTGRES:TABLE:2 \& [ERRORS TABLE COUNT] \& OK, Oracle and PostgreSQL have the same number of TABLE. \& \& [TEST TABLE TRIGGERS COUNT] \& ORACLEDB:DEPARTMENTS:0 \& POSTGRES:departments:0 \& ORACLEDB:EMPLOYEES:1 \& POSTGRES:employees:1 \& [ERRORS TABLE TRIGGERS COUNT] \& OK, Oracle and PostgreSQL have the same number of table triggers. \& \& [TEST TRIGGER COUNT] \& ORACLEDB:TRIGGER:2 \& POSTGRES:TRIGGER:2 \& [ERRORS TRIGGER COUNT] \& OK, Oracle and PostgreSQL have the same number of TRIGGER. \& \& [TEST VIEW COUNT] \& ORACLEDB:VIEW:1 \& POSTGRES:VIEW:1 \& [ERRORS VIEW COUNT] \& OK, Oracle and PostgreSQL have the same number of VIEW. \& \& [TEST MVIEW COUNT] \& ORACLEDB:MVIEW:0 \& POSTGRES:MVIEW:0 \& [ERRORS MVIEW COUNT] \& OK, Oracle and PostgreSQL have the same number of MVIEW. \& \& [TEST SEQUENCE COUNT] \& ORACLEDB:SEQUENCE:1 \& POSTGRES:SEQUENCE:0 \& [ERRORS SEQUENCE COUNT] \& SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). \& \& [TEST TYPE COUNT] \& ORACLEDB:TYPE:1 \& POSTGRES:TYPE:0 \& [ERRORS TYPE COUNT] \& TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). \& \& [TEST FDW COUNT] \& ORACLEDB:FDW:0 \& POSTGRES:FDW:0 \& [ERRORS FDW COUNT] \& OK, Oracle and PostgreSQL have the same number of FDW. \& \& [TEST FUNCTION COUNT] \& ORACLEDB:FUNCTION:3 \& POSTGRES:FUNCTION:3 \& [ERRORS FUNCTION COUNT] \& OK, Oracle and PostgreSQL have the same number of functions. \& \& [TEST SEQUENCE VALUES] \& ORACLEDB:EMPLOYEES_NUM_SEQ:1285 \& POSTGRES:employees_num_seq:1285 \& [ERRORS SEQUENCE VALUES COUNT] \& OK, Oracle and PostgreSQL have the same values for sequences \& \& [TEST ROWS COUNT] \& ORACLEDB:DEPARTMENTS:27 \& POSTGRES:departments:27 \& ORACLEDB:EMPLOYEES:854 \& POSTGRES:employees:854 \& [ERRORS ROWS COUNT] \& OK, Oracle and PostgreSQL have the same number of rows. .Ve .SS "Data validation" .IX Subsection "Data validation" Data validation consists in comparing data retrieved from a foreign table pointing to the source Oracle table and a local PostgreSQL table resulting from the data export. .PP To run data validation you can use a direct connection like any other Ora2Pg action but you can also use the oracle_fdw, mysql_fdw ior tds_fdw extension provided that \s-1FDW_SERVER\s0 and \s-1PG_DSN\s0 configuration directives are set. .PP By default Ora2Pg will extract the 10000 first rows from both side, you can change this value using directive \s-1DATA_VALIDATION_ROWS.\s0 When it is set to zero all rows of the tables will be compared. .PP Data validation requires that the table has a primary key or unique index and that the key columns is not a \s-1LOB.\s0 Rows will be sorted using this unique key. Due to differences in sort behavior between Oracle and PostgreSQL, if the collation of unique key columns in PostgreSQL is not 'C', the sort order can be different compared to Oracle. In this case the data validation will fail. .PP Data validation must be done before any data is modified. .PP Ora2Pg will stop comparing two tables after \s-1DATA_VALIDATION_ROWS\s0 is reached or that 10 errors has been encountered, result is dumped in a file named \&\*(L"data_validation.log\*(R" written in the current directory by default. The number of error before stopping the diff between rows can be controlled using the configuration directive \s-1DATA_VALIDATION_ERROR.\s0 All rows in errors are printed to the output file for your analyze. .PP It is possible to parallelize data validation by using \-P option or the corresponding configuration directive \s-1PARALLEL_TABLES\s0 in ora2pg.conf. .SS "Use of System Change Number (\s-1SCN\s0)" .IX Subsection "Use of System Change Number (SCN)" Ora2Pg is able to export data as of a specific \s-1SCN.\s0 You can set it at command line using the \-S or \-\-scn option. You can give a specific \s-1SCN\s0 or if you want to use the current \s-1SCN\s0 at first connection time set the value to 'current'. In this last case the connection user has the \*(L"\s-1SELECT ANY DICTIONARY\*(R"\s0 or the \&\*(L"\s-1SELECT_CATALOG_ROLE\*(R"\s0 role, the current \s-1SCN\s0 is looked at the v$database view. .PP Example of use: .PP .Vb 1 \& ora2pg \-c ora2pg.conf \-t COPY \-\-scn 16605281 .Ve .PP This adds the following clause to the query used to retrieve data for example: .PP .Vb 1 \& AS OF SCN 16605281 .Ve .PP You can also use th \-\-scn option to use the Oracle flashback capabality by specifying a timestamp expression instead of a \s-1SCN.\s0 For example: .PP .Vb 1 \& ora2pg \-c ora2pg.conf \-t COPY \-\-scn "TO_TIMESTAMP(\*(Aq2021\-12\-01 00:00:00\*(Aq, \*(AqYYYY\-MM\-DD HH:MI:SS\*(Aq)" .Ve .PP This will add the following clause to the query used to retrieve data: .PP .Vb 1 \& AS OF TIMESTAMP TO_TIMESTAMP(\*(Aq2021\-12\-01 00:00:00\*(Aq, \*(AqYYYY\-MM\-DD HH:MI:SS\*(Aq) .Ve .PP or for example to only retrive yesterday's data: .PP .Vb 1 \& ora2pg \-c ora2pg.conf \-t COPY \-\-scn "SYSDATE \- 1" .Ve .SS "Change Data Capture (\s-1CDC\s0)" .IX Subsection "Change Data Capture (CDC)" Ora2Pg do not have such feature which allow to import data and to only apply changes after the first import. But you can use the \-\-cdc_ready option to export data with registration of the \s-1SCN\s0 at the time of the table export. All \s-1SCN\s0 per tables are written to a file named \s-1TABLES_SCN\s0.log by default, it can be changed using \-C | \-\-cdc_file option. .PP These \s-1SCN\s0 registered per table during \s-1COPY\s0 or \s-1INSERT\s0 export can be used with a \s-1CDC\s0 tool. The format of the file is tablename:SCN per line. .SS "Importing \s-1BLOB\s0 as large objects" .IX Subsection "Importing BLOB as large objects" By default Ora2Pg imports Oracle \s-1BLOB\s0 as bytea, the destination column is created using the bytea data type. If you want to use large object instead of bytea, just add the \-\-blob_to_lo option to the ora2pg command. It will create the destination column as data type Oid and will save the \s-1BLOB\s0 as a large object using the \fBlo_from_bytea()\fR function. The Oid returned by the call to \fBlo_from_bytea()\fR is inserted in the destination column instead of a bytea. Because of the use of the function this option can only be used with actions \s-1SHOW_COLUMN, TABLE\s0 and \s-1INSERT.\s0 Action \s-1COPY\s0 is not allowed. .PP If you want to use \s-1COPY\s0 or have huge size \s-1BLOB\s0 ( > 1GB) than can not be imported using \fBlo_from_bytea()\fR you can add option \-\-lo_import to the ora2pg command. This will allow to import data in two passes. .PP 1) Export data using \s-1COPY\s0 or \s-1INSERT\s0 will set the Oid destination column for \&\s-1BLOB\s0 to value 0 and save the \s-1BLOB\s0 value into a dedicated file. It will also create a Shell script to import the \s-1BLOB\s0 files into the database using psql command \elo_import and to update the table Oid column to the returned large object Oid. The script is named lo_import\-TABLENAME.sh .PP 2) Execute all scripts lo_import\-TABLENAME.sh after setting the environment variables \s-1PGDATABASE\s0 and optionally \s-1PGHOST, PGPORT, PGUSER,\s0 etc. if they do not correspond to the default values for libpq. .PP You might also execute manually a \s-1VACUUM FULL\s0 on the table to remove the bloat created by the table update. .PP Limitation: the table must have a primary key, it is used to set the \s-1WHERE\s0 clause to update the Oid column after the large object import. Importing \s-1BLOB\s0 using this second method (\-\-lo_import) is very slow so it should be reserved to rows where the \s-1BLOB\s0 > 1GB for all other rows use the option \-\-blob_to_lo. To filter the rows you can use the \s-1WHERE\s0 configuration directive in ora2pg.conf. .SH "SUPPORT" .IX Header "SUPPORT" .SS "Author / Maintainer" .IX Subsection "Author / Maintainer" Gilles Darold .PP Please report any bugs, patches, help, etc. to . .SS "Feature request" .IX Subsection "Feature request" If you need new features let me know at . This help a lot to develop a better/useful tool. .SS "How to contribute ?" .IX Subsection "How to contribute ?" Any contribution to build a better tool is welcome, you just have to send me your ideas, features request or patches and there will be applied. .SH "LICENSE" .IX Header "LICENSE" Copyright (c) 2000\-2024 Gilles Darold \- All rights reserved. .PP .Vb 4 \& This program is free software: you can redistribute it and/or modify \& it under the terms of the GNU General Public License as published by \& the Free Software Foundation, either version 3 of the License, or \& any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You should have received a copy of the GNU General Public License \& along with this program. If not, see < http://www.gnu.org/licenses/ >. .Ve .SH "ACKNOWLEDGEMENT" .IX Header "ACKNOWLEDGEMENT" I must thanks a lot all the great contributors, see changelog for all acknowledgments. ora2pg-25.0/lib/000077500000000000000000000000001500113072400133715ustar00rootroot00000000000000ora2pg-25.0/lib/Ora2Pg.pm000066400000000000000000027117201500113072400150330ustar00rootroot00000000000000package Ora2Pg; #------------------------------------------------------------------------------ # Project : Oracle to PostgreSQL database schema converter # Name : Ora2Pg.pm # Language : Perl # Authors : Gilles Darold, gilles _AT_ darold _DOT_ net # Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved - # Function : Main module used to export Oracle database schema to PostgreSQL # Usage : See documentation in this file with perldoc. #------------------------------------------------------------------------------ # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see < http://www.gnu.org/licenses/ >. # #------------------------------------------------------------------------------ use vars qw($VERSION $PSQL %AConfig); use Carp qw(confess); use DBI; use POSIX qw(locale_h _exit :sys_wait_h strftime); use IO::File; use Config; use Time::HiRes qw/usleep/; use Fcntl qw/ :flock /; use IO::Handle; use IO::Pipe; use File::Basename; use File::Spec; use File::Temp qw/ tempfile /; use Benchmark; use Encode; #set locale to LC_NUMERIC C setlocale(LC_NUMERIC,"C"); $VERSION = '25.0'; $PSQL = $ENV{PLSQL} || 'psql'; $| = 1; our %RUNNING_PIDS = (); # Multiprocess communication pipe our $pipe = undef; our $TMP_DIR = File::Spec->tmpdir() || '/tmp'; our %ordered_views = (); # Character that must be escaped in COPY statement my $ESCAPE_COPY = { "\0" => "", "\\" => "\\\\", "\r" => "\\r", "\n" => "\\n", "\t" => "\\t"}; # Oracle internal timestamp month equivalent our %ORACLE_MONTHS = ('JAN'=>'01', 'FEB'=>'02','MAR'=>'03','APR'=>'04','MAY'=>'05','JUN'=>'06','JUL'=>'07','AUG'=>'08','SEP'=>'09','OCT'=>10,'NOV'=>11,'DEC'=>12); # Exclude table generated by partition logging, materialized view logs, statistis on spatial index, # spatial index tables, sequence index tables, interMedia Text index tables and Unified Audit tables. # LogMiner, Oracle Advanced Replication, hash table used by loadjava. our @EXCLUDED_TABLES = ('USLOG\$_.*', 'MLOG\$_.*', 'RUPD\$_.*', 'MDXT_.*', 'MDRT_.*', 'MDRS_.*', 'DR\$.*', 'CLI_SWP\$.*', 'LOGMNR\$.*', 'REPCAT\$.*', 'JAVA\$.*', 'AQ\$.*', 'BIN\$.*', 'SDO_GR_.*', '.*\$JAVA\$.*', 'PROF\$.*', 'TOAD_PLAN_.*', 'SYS_.*\$', 'QUEST_SL_.*', 'SYS_EXPORT_SCHEMA_.*', 'SYS_IMPORT_.*'); our @EXCLUDED_TABLES_8I = ('USLOG$_%', 'MLOG$_%', 'RUPD$_%', 'MDXT_%', 'MDRT_%', 'MDRS_%', 'DR$%', 'CLI_SWP$%', 'LOGMNR$%', 'REPCAT$%', 'JAVA$%', 'AQ$%', 'BIN$%', '%$JAVA$%', 'PROF$%', 'TOAD_PLAN_%', 'SYS_%$', 'QUEST_SL_%', 'SYS_EXPORT_SCHEMA_%', 'SYS_IMPORT_%'); our @Oracle_tables = qw( EVT_CARRIER_CONFIGURATION EVT_DEST_PROFILE EVT_HISTORY EVT_INSTANCE EVT_MAIL_CONFIGURATION EVT_MONITOR_NODE EVT_NOTIFY_STATUS EVT_OPERATORS EVT_OPERATORS_ADDITIONAL EVT_OPERATORS_SYSTEMS EVT_OUTSTANDING EVT_PROFILE EVT_PROFILE_EVENTS EVT_REGISTRY EVT_REGISTRY_BACKLOG OLS_DIR_BUSINESSE OLS_DIR_BUSINESSES SDO_COORD_REF_SYS SDO_CS_SRS SDO_INDEX_METADATA_TABLE SDO_INDEX_METADATA_TABLES SDO_PC_BLK_TABLE SDO_STYLES_TABLE SDO_TIN_BLK_TABLE SMACTUALPARAMETER_S SMGLOBALCONFIGURATION_S SMFORMALPARAMETER_S SMFOLDER_S SMDISTRIBUTIONSET_S SMDEPENDENTLINKS SMDEPENDENTINDEX SMDEPENDEEINDEX SMDEFAUTH_S SMDBAUTH_S SMPARALLELJOB_S SMPACKAGE_S SMOWNERLINKS SMOWNERINDEX SMOWNEEINDEX SMOSNAMES_X SMOMSTRING_S SMMOWNERLINKS SMMOWNERINDEX SMPACKAGE_S SMPARALLELJOB_S SMPARALLELOPERATION_S SMPARALLELSTATEMENT_S SMPRODUCT_S SMP_AD_ADDRESSES_ SMP_AD_DISCOVERED_NODES_ SMP_AD_NODES_ SMP_AD_PARMS_ SMP_AUTO_DISCOVERY_ITEM_ SMP_AUTO_DISCOVERY_PARMS_ SMP_BLOB_ SMP_CREDENTIALS\$ SMP_JOB_ SMP_JOB_EVENTLIST_ SMP_JOB_HISTORY_ SMP_JOB_INSTANCE_ SMP_JOB_LIBRARY_ SMP_JOB_TASK_INSTANCE_ SMP_LONG_TEXT_ SMP_REP_VERSION SMP_SERVICES SMP_SERVICE_GROUP_DEFN_ SMP_SERVICE_GROUP_ITEM_ SMP_SERVICE_ITEM_ SMP_UPDATESERVICES_CALLED_ SMAGENTJOB_S SMARCHIVE_S SMBREAKABLELINKS SMCLIQUE SMCONFIGURATION SMCONSOLESOSETTING_S SMDATABASE_S SMHOSTAUTH_S SMHOST_S SMINSTALLATION_S SMLOGMESSAGE_S SMMONTHLYENTRY_S SMMONTHWEEKENTRY_S SMP_USER_DETAILS SMRELEASE_S SMRUN_S SMSCHEDULE_S SMSHAREDORACLECLIENT_S SMSHAREDORACLECONFIGURATION_S SMTABLESPACE_S SMVCENDPOINT_S SMWEEKLYENTRY_S ); push(@EXCLUDED_TABLES, @Oracle_tables); # Some function might be excluded from export and assessment. our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); our @FKEY_OPTIONS = ('NEVER', 'DELETE', 'ALWAYS'); # Minimized the footprint on disc, so that more rows fit on a data page, # which is the most important factor for speed. our %TYPALIGN = ( # Types and size, 1000 = variable 'boolean' => 1, 'smallint' => 2, 'smallserial' => 2, 'integer' => 4, 'real' => 4, 'serial' => 4, 'date' => 4, 'oid' => 4, 'macaddr' => 6, 'bigint' => 8, 'bigserial' => 8, 'double precision' => 8, 'macaddr8' => 8, 'money' => 8, 'time' => 8, 'timestamp' => 8, 'timestamp without time zone' => 8, 'timestamp with time zone' => 8, 'interval' => 16, 'point' => 16, 'tinterval' => 16, 'uuid' => 16, 'circle' => 24, 'box' => 32, 'line' => 32, 'lseg' => 32, 'bit' => 1000, 'bytea' => 1000, 'character varying' => 1000, 'cidr' => 19, 'json' => 1000, 'jsonb' => 1000, 'numeric' => 1000, 'path' => 1000, 'polygon' => 1000, 'text' => 1000, 'xml' => 1000, # aliases 'bool' => 1, 'timetz' => 12, 'char' => 1000, 'decimal' => 1000, # deprecated 'int2' => 2, 'abstime' => 4, 'bpchar' => 4, 'int4' => 4, 'reltime' => 4, 'float4' => 4, 'timestamptz' => 8, 'float8' => 8, 'int8' => 8, 'name' => 64, 'inet' => 19, 'varbit' => 1000, 'varchar' => 1000 ); our %INDEX_TYPE = ( 'NORMAL' => 'b-tree', 'NORMAL/REV' => 'reversed b-tree', 'FUNCTION-BASED NORMAL' => 'function based b-tree', 'FUNCTION-BASED NORMAL/REV' => 'function based reversed b-tree', 'BITMAP' => 'bitmap', 'BITMAP JOIN' => 'bitmap join', 'FUNCTION-BASED BITMAP' => 'function based bitmap', 'FUNCTION-BASED BITMAP JOIN' => 'function based bitmap join', 'CLUSTER' => 'cluster', 'DOMAIN' => 'domain', 'IOT - TOP' => 'IOT', 'SPATIAL INDEX' => 'spatial index', ); # Reserved keywords in PostgreSQL our @KEYWORDS = qw( ALL ANALYSE ANALYZE AND ANY ARRAY AS ASC ASYMMETRIC AUTHORIZATION BINARY BOTH CASE CAST CHECK CMAX CMIN COLLATE COLLATION COLUMN CONCURRENTLY CONSTRAINT CREATE CROSS CTID CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER DEFAULT DEFERRABLE DESC DISTINCT DO ELSE END EXCEPT FALSE FETCH FOR FOREIGN FREEZE FROM FULL GRANT GROUP HAVING ILIKE IN INITIALLY INNER INTERSECT INTO IS ISNULL JOIN KEY LATERAL LEADING LEFT LIKE LIMIT LOCALTIME LOCALTIMESTAMP NATURAL NOT NOTNULL NULL OFFSET ON ONLY OR ORDER OUTER OVERLAPS PARTITION PLACING PRIMARY REFERENCES REF RETURNING RIGHT SELECT SESSION_USER SIMILAR SOME SYMMETRIC TABLE TABLESAMPLE THEN TO TRAILING TRUE UNION UNIQUE USER USING VARIADIC VERBOSE WHEN WHERE WINDOW WITH ); # Reserved keywords that can be used in PostgreSQL as function or type name our @FCT_TYPE_KEYWORDS = qw( AUTHORIZATION BINARY COLLATION CONCURRENTLY CROSS CURRENT_SCHEMA FREEZE FULL ILIKE INNER IS ISNULL JOIN LEFT LIKE NATURAL NOTNULL OUTER OVERLAPS RIGHT SIMILAR TABLESAMPLE VERBOSE ); our @SYSTEM_FIELDS = qw(oid tableoid xmin xmin cmin xmax cmax ctid); our %BOOLEAN_MAP = ( 'yes' => 't', 'no' => 'f', 'y' => 't', 'n' => 'f', '1' => 't', '0' => 'f', 'true' => 't', 'false' => 'f', 'enabled'=> 't', 'disabled'=> 'f', 't' => 't', 'f' => 'f', ); our @GRANTS = ( 'SELECT', 'INSERT', 'UPDATE', 'DELETE', 'TRUNCATE', 'REFERENCES', 'TRIGGER', 'USAGE', 'CREATE', 'CONNECT', 'TEMPORARY', 'TEMP', 'USAGE', 'ALL', 'ALL PRIVILEGES', 'EXECUTE' ); our @ORACLE_FDW_COPY_MODES = qw( local server ); our @ORACLE_FDW_COPY_FORMATS = qw( binary csv ); $SIG{'CHLD'} = 'DEFAULT'; #### # method used to fork as many child as wanted ## sub spawn { my $coderef = shift; unless (@_ == 0 && $coderef && ref($coderef) eq 'CODE') { print "usage: spawn CODEREF"; exit 0; } my $pid; if (!defined($pid = fork)) { print STDERR "Error: cannot fork: $!\n"; return; } elsif ($pid) { $RUNNING_PIDS{$pid} = $pid; return; # the parent } # the child -- go spawn $< = $>; $( = $); # suid progs only exit &$coderef(); } # With multiprocess we need to wait all childs sub wait_child { my $sig = shift; print STDERR "Received terminating signal ($sig).\n"; if ($^O !~ /MSWin32|dos/i) { 1 while wait != -1; $SIG{INT} = \&wait_child; $SIG{TERM} = \&wait_child; } print STDERR "Aborting.\n"; _exit(0); } $SIG{INT} = \&wait_child; $SIG{TERM} = \&wait_child; =head1 PUBLIC METHODS =head2 new HASH_OPTIONS Creates a new Ora2Pg object. The only required option is: - config : Path to the configuration file (required). All directives found in the configuration file can be overwritten in the instance call by passing them in lowercase as arguments. =cut sub new { my ($class, %options) = @_; # This create an OO perl object my $self = {}; bless ($self, $class); # Initialize this object $self->_init(%options); # Return the instance return($self); } =head2 export_schema FILENAME Print SQL data output to a file name or to STDOUT if no file name is specified. =cut sub export_schema { my $self = shift; # Create default export file where things will be written with the dump() method # First remove it if the output file already exists foreach my $t (@{$self->{export_type}}) { next if ($t =~ /^(?:SHOW_|TEST)/i); # SHOW_* commands are not concerned here # Set current export type $self->{type} = $t; if ($self->{type} ne 'LOAD') { # Close open main output file if (defined $self->{fhout}) { $self->close_export_file($self->{fhout}); } # Remove old export file if it already exists $self->remove_export_file(); # then create a new one $self->create_export_file(); } # Dump exported statement to output $self->_get_sql_statements(); if ($self->{type} ne 'LOAD') { # Close output export file create above $self->close_export_file($self->{fhout}) if (defined $self->{fhout}); } } # Disconnect from the database $self->{dbh}->disconnect() if ($self->{dbh}); $self->{dbhdest}->disconnect() if ($self->{dbhdest}); # Try to requalify package function call if (!$self->{package_as_schema}) { $self->fix_function_call(); } my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); unlink($dirprefix . 'temp_pass2_file.dat'); if ($self->{type} eq 'COPY' && !$self->{quiet}) { print "\nSchema Export Complete\n\n"; } } =head2 open_export_file FILENAME Open a file handle to a given filename. =cut sub open_export_file { my ($self, $outfile, $noprefix) = @_; my $filehdl = undef; if ($outfile && $outfile ne '-') { if ($outfile ne '-') { if ($self->{output_dir} && !$noprefix) { $outfile = $self->{output_dir} . '/' . $outfile; } if ($self->{input_file} && ($outfile eq $self->{input_file})) { $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); } } # If user request data compression if ($outfile =~ /\.gz$/i) { eval("use Compress::Zlib;"); $self->{compress} = 'Zlib'; $filehdl = gzopen("$outfile", "wb") or $self->logit("FATAL: Can't create deflation file $outfile\n",0,1); if ($self->{'binmode'} =~ /^:/) { binmode($filehdl, $self->{'binmode'}); } else { binmode($filehdl, ":utf8"); } } elsif ($outfile =~ /\.bz2$/i) { $self->logit("Error: can't run bzip2\n",0,1) if (!-x $self->{bzip2}); $self->{compress} = 'Bzip2'; $filehdl = new IO::File; $filehdl->open("|$self->{bzip2} --stdout >$outfile") or $self->logit("FATAL: Can't open pipe to $self->{bzip2} --stdout >$outfile: $!\n", 0,1); if ($self->{'binmode'} =~ /^:/) { binmode($filehdl, $self->{'binmode'}); } else { binmode($filehdl, ":utf8"); } } else { $filehdl = new IO::File; $filehdl->open(">$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); } $filehdl->autoflush(1) if (defined $filehdl && !$self->{compress}); } return $filehdl; } =head2 create_export_file FILENAME Set output file and open a file handle on it, will use STDOUT if no file name is specified. =cut sub create_export_file { my ($self, $outfile) = @_; # Do not create the default export file with direct data export if (($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY')) { return if ($self->{pg_dsn}); } # Init with configuration OUTPUT filename $outfile ||= $self->{output}; if ($outfile) { if ($outfile ne '-') { # Prefix out file with export type in multiple export type call $outfile = $self->{type} . "_$outfile" if ($#{$self->{export_type}} > 0); if ($self->{output_dir} && $outfile) { $outfile = $self->{output_dir} . "/" . $outfile; } if ($self->{input_file} && ($outfile eq $self->{input_file})) { $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); } } # Send output to the specified file if ($outfile =~ /\.gz$/) { eval("use Compress::Zlib;"); $self->{compress} = 'Zlib'; $self->{fhout} = gzopen($outfile, "wb") or $self->logit("FATAL: Can't create deflation file $outfile\n", 0, 1); } elsif ($outfile =~ /\.bz2$/) { $self->logit("FATAL: can't run bzip2\n",0,1) if (!-x $self->{bzip2}); $self->{compress} = 'Bzip2'; $self->{fhout} = new IO::File; $self->{fhout}->open("|$self->{bzip2} --stdout >$outfile") or $self->logit("FATAL: Can't open pipe to $self->{bzip2} --stdout >$outfile: $!\n", 0, 1); } else { $self->{fhout} = new IO::File; $self->{fhout}->open(">>$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); $self->set_binmode($self->{fhout}); } if ( $self->{compress} && (($self->{jobs} > 1) || ($self->{oracle_copies} > 1)) ) { die "FATAL: you can't use compressed output with parallel dump\n"; } } } sub remove_export_file { my ($self, $outfile) = @_; # Init with configuration OUTPUT filename $outfile ||= $self->{output}; if ($outfile && $outfile ne '-') { # Prefix out file with export type in multiple export type call $outfile = $self->{type} . "_$outfile" if ($#{$self->{export_type}} > 0); if ($self->{output_dir} && $outfile) { $outfile = $self->{output_dir} . "/" . $outfile; } if ($self->{input_file} && ($outfile eq $self->{input_file})) { $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); } unlink($outfile); } } =head2 append_export_file FILENAME Open a file handle to a given filename to append data. =cut sub append_export_file { my ($self, $outfile, $noprefix) = @_; my $filehdl = undef; if ($outfile) { if ($self->{output_dir} && !$noprefix) { $outfile = $self->{output_dir} . '/' . $outfile; } # If user request data compression if ($self->{compress} && (($self->{jobs} > 1) || ($self->{oracle_copies} > 1))) { die "FATAL: you can't use compressed output with parallel dump\n"; } else { $filehdl = new IO::File; $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); $filehdl->autoflush(1); } } return $filehdl; } =head2 append_lo_import_file FILENAME Open a file handle to lo_import-$table.sh file to append data. =cut sub append_lo_import_file { my ($self, $table, $noprefix) = @_; my $filehdl = undef; my $new = 0; my $outfile = "lo_import-$table.sh"; if ($self->{output_dir} && !$noprefix) { $outfile = $self->{output_dir} . '/' . $outfile; } $filehdl = new IO::File; $new = 1 if (!-e $outfile); $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); $filehdl->autoflush(1); $self->set_binmode($filehdl); flock($filehdl, 2) || die "FATAL: can't lock file $outfile\n"; # At file creation append the verification of the required environment variable if ($new) { $self->{post_lo_script} = qq{#!/bin/sh if [ "\${PGDATABASE}a" = "a" ]; then echo "You must set the environment variable PGDATABASE to defined the database where the commands" echo "will be executed. And optionally PGHOST, PGUSER, etc. if they don't correspond to the default." exit 1 fi $self->{post_lo_script} }; } $filehdl->print($self->{post_lo_script}); $filehdl->close(); } =head2 read_export_file FILENAME Open a file handle to a given filename to read data. =cut sub read_export_file { my ($self, $infile) = @_; my $filehdl = new IO::File; $filehdl->open("<$infile") or $self->logit("FATAL: Can't read $infile: $!\n", 0, 1); return $filehdl; } =head2 close_export_file FILEHANDLE Close a file handle. =cut sub close_export_file { my ($self, $filehdl, $not_compressed) = @_; return if (!defined $filehdl); if (!$not_compressed && $self->{output} =~ /\.gz$/) { if ($filehdl =~ /IO::File=/) { $filehdl->close(); } else { $filehdl->gzclose(); } } else { $filehdl->close(); } } =head2 modify_struct TABLE_NAME ARRAYOF_FIELDNAME Modify the table structure during the export. Only the specified columns will be exported. =cut sub modify_struct { my ($self, $table, @fields) = @_; if (!$self->{preserve_case}) { map { $_ = lc($_) } @fields; $table = lc($table); } push(@{$self->{modify}{$table}}, @fields); } =head2 exclude_columns TABLE_NAME ARRAYOF_FIELDNAME Modify the table structure during the export. The specified columns will NOT be exported. =cut sub exclude_columns { my ($self, $table, @fields) = @_; if (!$self->{preserve_case}) { delete $self->{exclude_columns}{$table}; map { $_ = lc($_) } @fields; $table = lc($table); push(@{$self->{exclude_columns}{$table}}, @fields); } } =head2 is_reserved_words Returns 1 if the given object name is a PostgreSQL reserved word Returns 2 if the object name is only numeric Returns 3 if the object name is a system column =cut sub is_reserved_words { my ($self, $obj_name) = @_; if ($obj_name && grep(/^\Q$obj_name\E$/i, @KEYWORDS)) { return 1 if (!grep(/^$self->{type}/, 'FUNCTION', 'PACKAGE', 'PROCEDURE') || grep(/^\Q$obj_name\E$/i, @FCT_TYPE_KEYWORDS)); } # columns starting by numbers need to be quoted unless it is an operation if ($obj_name =~ /^\d+/ && $obj_name !~ /[\+\-\/\*]/) { return 2; } if ($obj_name && grep(/^\Q$obj_name\E$/i, @SYSTEM_FIELDS)) { return 3; } return 0; } =head2 quote_object_name Return a quoted object named when needed: - PostgreSQL reserved word - unsupported character - start with a digit or digit only =cut sub quote_object_name { my ($self, @obj_list) = @_; my @ret = (); foreach my $obj_name (@obj_list) { next if ($obj_name =~ /^SYS_NC\d+/); # Start by removing any double quote and extra space $obj_name =~ s/"//g; $obj_name =~ s/^\s+//; $obj_name =~ s/\s+$//; # When PRESERVE_CASE is not enabled set object name to lower case if (!$self->{preserve_case}) { $obj_name = lc($obj_name); # then if there is non alphanumeric or the object name is a reserved word if ($obj_name =~ /[^a-z0-9\_\.\$]/ || ($self->{use_reserved_words} && $self->is_reserved_words($obj_name)) || $obj_name =~ /^\d+/) { # Add double quote to [schema.] object name if ($obj_name !~ /^[^\.]+\.[^\.]+$/ && $obj_name !~ /^[^\.]+\.[^\.]+\.[^\.]+$/) { $obj_name = '"' . $obj_name . '"'; } elsif ($obj_name =~ /^[^\.]+\.[^\.]+$/) { $obj_name =~ s/^([^\.]+)\.([^\.]+)$/"$1"\."$2"/; } else { $obj_name =~ s/^([^\.]+)\.([^\.]+)\.([^\.]+)$/"$1"\."$2"\."$3"/; } $obj_name = '"' . $obj_name . '"' if ($obj_name =~ /^\d+/); } } # Add double quote to [schema.] object name elsif ($obj_name !~ /^[^\.]+\.[^\.]+$/ && $obj_name !~ /^[^\.]+\.[^\.]+\.[^\.]+$/) { $obj_name = "\"$obj_name\""; } elsif ($obj_name =~ /^[^\.]+\.[^\.]+$/) { $obj_name =~ s/^([^\.]+)\.([^\.]+)$/"$1"\."$2"/; } else { $obj_name =~ s/^([^\.]+)\.([^\.]+)\.([^\.]+)$/"$1"\."$2"\."$3"/; } if ($obj_name =~ /^"[^\s]+\s+(ASC|DESC)"$/i) { $obj_name =~ s/"//g; $obj_name =~ s/\s+ASC$//ig; } push(@ret, $obj_name); } return join(',', @ret); } =head2 replace_tables HASH Modify table names during the export. =cut sub replace_tables { my ($self, %tables) = @_; foreach my $t (keys %tables) { $self->{replaced_tables}{"\L$t\E"} = $tables{$t}; } } =head2 replace_cols HASH Modify column names during the export. =cut sub replace_cols { my ($self, %cols) = @_; foreach my $t (keys %cols) { foreach my $c (keys %{$cols{$t}}) { $self->{replaced_cols}{"\L$t\E"}{"\L$c\E"} = $cols{$t}{$c}; } } } =head2 set_where_clause HASH Add a WHERE clause during data export on specific tables or on all tables =cut sub set_where_clause { my ($self, $global, %table_clause) = @_; $self->{global_where} = $global; foreach my $t (keys %table_clause) { $self->{where}{"\L$t\E"} = $table_clause{$t}; } } =head2 set_delete_clause HASH Add a DELETE clause before data export on specific tables or on all tables =cut sub set_delete_clause { my ($self, $global, %table_clause) = @_; $self->{global_delete} = $global; foreach my $t (keys %table_clause) { $self->{delete}{"\L$t\E"} = $table_clause{$t}; } } #### Private subroutines #### =head1 PRIVATE METHODS =head2 _db_connection Initialize a connexion to the Oracle database. =cut sub _db_connection { my $self = shift; if ($self->{is_mysql}) { use Ora2Pg::MySQL; return Ora2Pg::MySQL::_db_connection($self); } elsif ($self->{is_mssql}) { use Ora2Pg::MSSQL; return Ora2Pg::MSSQL::_db_connection($self); } else { use Ora2Pg::Oracle; return Ora2Pg::Oracle::_db_connection($self); } } =head2 _init HASH_OPTIONS Initialize an Ora2Pg object instance with a connexion to the Oracle database. =cut sub _init { my ($self, %options) = @_; # Use custom temp directory if specified $TMP_DIR = $options{temp_dir} || $TMP_DIR; # Read configuration file $self->read_config($options{config}) if ($options{config}); # Override any Ora2Pg configuration directive from command line options if (exists $options{options}) { my @opt = split(/\s*\|\s*/, $options{options}); foreach my $o (@opt) { next if ($o =~ /^options\s*=/i); $o =~ s/\s*=\s*/\t/; &parse_config($o); } delete $options{options}; } # Those are needed by DBI $ENV{ORACLE_HOME} = $AConfig{'ORACLE_HOME'} if ($AConfig{'ORACLE_HOME'}); $ENV{NLS_LANG} = $AConfig{'NLS_LANG'} if ($AConfig{'NLS_LANG'}); # Init arrays $self->{default_tablespaces} = (); $self->{limited} = (); $self->{excluded} = (); $self->{view_as_table} = (); $self->{mview_as_table} = (); $self->{modify} = (); $self->{replaced_tables} = (); $self->{replaced_cols} = (); $self->{replace_as_boolean} = (); $self->{ora_boolean_values} = (); $self->{null_equal_empty} = 1; $self->{estimate_cost} = 0; $self->{where} = (); $self->{delete} = (); $self->{replace_query} = (); $self->{ora_reserved_words} = (); $self->{defined_pk} = (); $self->{allow_partition} = (); $self->{empty_lob_null} = 0; $self->{look_forward_function} = (); $self->{no_function_metadata} = 0; $self->{transform_value} = (); $self->{all_objects} = (); # Initial command to execute at Oracle and PostgreSQL connexion $self->{ora_initial_command} = (); $self->{pg_initial_command} = (); # To register user defined exception $self->{custom_exception} = (); $self->{exception_id} = 50001; # Init PostgreSQL DB handle $self->{dbhdest} = undef; $self->{standard_conforming_strings} = 1; $self->{create_schema} = 1; # Init some arrays $self->{external_table} = (); $self->{function_metadata} = (); $self->{grant_object} = ''; # Used to precise if we need to rename the partitions $self->{rename_partition} = 0; # Do not use parellel export for partition, backward compatibility with <= 25.0 $self->{disable_parallel_partition} = 0; # Use to preserve the data export type with geometry objects $self->{local_type} = ''; # Shall we log on error during data import or abort. $self->{log_on_error} = 0; # Initialize some variable related to export of mysql database $self->{is_mysql} = 0; $self->{mysql_mode} = ''; $self->{mysql_internal_extract_format} = 0; $self->{mysql_pipes_as_concat} = 0; # Initialize some variable related to export of mssql database $self->{is_mssql} = 0; $self->{drop_rowversion} = 0; $self->{case_insensitive_search} = 'citext'; # List of users for audit trail $self->{audit_user} = ''; # Disable copy freeze by default $self->{copy_freeze} = ''; # Use FTS index to convert CONTEXT Oracle's indexes by default $self->{context_as_trgm} = 0; $self->{fts_index_only} = 1; $self->{fts_config} = ''; $self->{use_unaccent} = 1; $self->{use_lower_unaccent} = 1; # Enable rewrite of outer join by default. $self->{rewrite_outer_join} = 1; # Init comment and text constant storage variables $self->{idxcomment} = 0; $self->{comment_values} = (); $self->{text_values} = (); $self->{text_values_pos} = 0; # Remove comments when reading an input file before parsing $self->{no_clean_comment} = 0; # Keep commit/rollback in converted pl/sql code by default $self->{comment_commit_rollback} = 0; # Keep savepoint in converted pl/sql code by default $self->{comment_savepoint} = 0; # Storage of string constant placeholder regexp $self->{string_constant_regexp} = (); $self->{alternative_quoting_regexp} = (); # Number of row for data validation $self->{data_validation_rows} = 10000; $self->{data_validation_error} = 10; $self->{data_validation_ordering} = 1; # Global file handle $self->{cfhout} = undef; # oracle_fdw foreign server $self->{fdw_server} = ''; # oracle_fdw copy mode $self->{oracle_fdw_copy_mode} = ''; # oracle_fdw copy format $self->{oracle_fdw_copy_format} = ''; # AS OF SCN related variables $self->{start_scn} = $options{start_scn} || ''; $self->{no_start_scn} = $options{no_start_scn} || ''; $self->{current_oracle_scn} = (); $self->{cdc_ready} = $options{cdc_ready} || ''; # Wether we load the pgtt extension as superuser or not $self->{pgtt_nosuperuser} ||= 0; # Initialyze following configuration file foreach my $k (sort keys %AConfig) { if (lc($k) eq 'allow') { $self->{limited} = $AConfig{ALLOW}; } elsif (lc($k) eq 'exclude') { $self->{excluded} = $AConfig{EXCLUDE}; } else { $self->{lc($k)} = $AConfig{$k}; } } # Set default tablespace to exclude when using USE_TABLESPACE push(@{$self->{default_tablespaces}}, 'TEMP', 'USERS','SYSTEM'); # Add the custom reserved keywords defined in configuration file push(@KEYWORDS, @{$self->{ora_reserved_words}}); # Verify grant objects if ($self->{type} eq 'GRANT' && $self->{grant_object}) { die "FATAL: wrong object type in GRANT_OBJECTS directive.\n" if (!grep(/^$self->{grant_object}$/, 'USER', 'TABLE', 'VIEW', 'MATERIALIZED VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE BODY', 'TYPE', 'SYNONYM', 'DIRECTORY')); } # Default boolean values foreach my $k (keys %BOOLEAN_MAP) { $self->{ora_boolean_values}{lc($k)} = $BOOLEAN_MAP{$k}; } # additional boolean values given from config file foreach my $k (keys %{$self->{boolean_values}}) { $self->{ora_boolean_values}{lc($k)} = $AConfig{BOOLEAN_VALUES}{$k}; } # Set transaction isolation level if ($self->{transaction} eq 'readonly') { $self->{transaction} = 'SET TRANSACTION READ ONLY'; } elsif ($self->{transaction} eq 'readwrite') { $self->{transaction} = 'SET TRANSACTION READ WRITE'; } elsif ($self->{transaction} eq 'committed') { $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED'; } elsif ($self->{transaction} eq 'serializable') { $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE'; } else { if (grep(/^$self->{type}$/, 'COPY', 'INSERT')) { $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE'; } else { $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED'; } } $self->{function_check} = 1 if (not defined $self->{function_check} || $self->{function_check} eq ''); $self->{qualify_function} = 1 if (!exists $self->{qualify_function}); # Set default function to use for uuid generation $self->{uuid_function} ||= 'uuid_generate_v4'; $self->{use_uuid} = 0; # Set default cost unit value to 5 minutes $self->{cost_unit_value} ||= 5; # Set default human days limit for type C migration level $self->{human_days_limit} ||= 5; # Defined if column order must be optimized $self->{reordering_columns} ||= 0; # Initialize suffix that may be added to the index name $self->{indexes_suffix} ||= ''; # Disable synchronous commit for pg data load $self->{synchronous_commit} ||= 0; # Disallow NOLOGGING / UNLOGGED table creation $self->{disable_unlogged} ||= 0; # Change the varchar max length value $self->{double_max_varchar} ||= 0; # Default degree for Oracle parallelism if ($self->{default_parallelism_degree} eq '') { $self->{default_parallelism_degree} = 0; } # For utf8 encoding of the stored procedure code $self->{force_plsql_encoding} ||= 0; # Add header to output file $self->{no_header} ||= 0; # Mark function as STABLE by default if (not defined $self->{function_stable} || $self->{function_stable} ne '0') { $self->{function_stable} = 1; } # Initialize rewriting of index name if (not defined $self->{indexes_renaming} || $self->{indexes_renaming} ne '0') { $self->{indexes_renaming} = 1; } # Enable autonomous transaction conversion. Default is enable it. if (!exists $self->{autonomous_transaction} || $self->{autonomous_transaction} ne '0') { $self->{autonomous_transaction} = 1; } # By default we force identity column to be bigint if (!exists $self->{force_identity_bigint} || $self->{force_identity_bigint} ne '0') { $self->{force_identity_bigint} = 1; } # by default we don't remove TZ part of the TO_CHAR() format if (!exists $self->{to_char_notimezone}) { $self->{to_char_notimezone} = 0; } # Don't use *_pattern_ops with indexes by default $self->{use_index_opclass} ||= 0; # Autodetect spatial type $self->{autodetect_spatial_type} ||= 0; # Use btree_gin extenstion to create bitmap like index with pg >= 9.4 $self->{bitmap_as_gin} = 1 if ($self->{bitmap_as_gin} ne '0'); # Create tables with OIDs or not, default to not create OIDs $self->{with_oid} ||= 0; # Minimum of lines required in a table to use parallelism $self->{parallel_min_rows} ||= 100000; # Should we export global temporary table $self->{export_gtt} ||= 0; # Should we replace zero date with something else than NULL $self->{replace_zero_date} ||= ''; if ($self->{replace_zero_date} && (uc($self->{replace_zero_date}) ne '-INFINITY') && ($self->{replace_zero_date} !~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/)) { die "FATAL: wrong format in REPLACE_ZERO_DATE value, should be YYYY-MM-DD HH:MM:SS or -INFINITY\n"; } # Defined default value for to_number translation $self->{to_number_conversion} ||= 'numeric'; # Set regexp to detect parts of statements that need to be considered as text if ($AConfig{STRING_CONSTANT_REGEXP}) { push(@{ $self->{string_constant_regexp} } , split(/;/, $AConfig{STRING_CONSTANT_REGEXP})); } if ($AConfig{ALTERNATIVE_QUOTING_REGEXP}) { push(@{ $self->{alternative_quoting_regexp} } , split(/;/, $AConfig{ALTERNATIVE_QUOTING_REGEXP})); } # Defined if we must add a drop if exists statement before creating an object $self->{drop_if_exists} ||= 0; # Disable ON CONFLICT clause by default $self->{insert_on_conflict} ||= 0; # Overwrite configuration with all given parameters # and try to preserve backward compatibility foreach my $k (keys %options) { next if ($options{options}); if (($k eq 'allow') && $options{allow}) { $self->{limited} = (); # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... my @allow_vlist = split(/\s*;\s*/, $options{allow}); foreach my $a (@allow_vlist) { if ($a =~ /^([^\[]+)\[(.*)\]$/) { push(@{$self->{limited}{"\U$1\E"}}, split(/[\s,]+/, $2) ); } else { push(@{$self->{limited}{ALL}}, split(/[\s,]+/, $a) ); } } } elsif (($k eq 'exclude') && $options{exclude}) { $self->{excluded} = (); # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... my @exclude_vlist = split(/\s*;\s*/, $options{exclude}); foreach my $a (@exclude_vlist) { if ($a =~ /^([^\[]+)\[(.*)\]$/) { push(@{$self->{excluded}{"\U$1\E"}}, split(/[\s,]+/, $2) ); } else { push(@{$self->{excluded}{ALL}}, split(/[\s,]+/, $a) ); } } } elsif (($k eq 'view_as_table') && $options{view_as_table}) { $self->{view_as_table} = (); push(@{$self->{view_as_table}}, split(/[\s;,]+/, $options{view_as_table}) ); } elsif (($k eq 'mview_as_table') && $options{mview_as_table}) { $self->{mview_as_table} = (); push(@{$self->{mview_as_table}}, split(/[\s;,]+/, $options{mview_as_table}) ); } elsif (($k eq 'datasource') && $options{datasource}) { $self->{oracle_dsn} = $options{datasource}; } elsif (($k eq 'user') && $options{user}) { $self->{oracle_user} = $options{user}; } elsif (($k eq 'password') && $options{password}) { $self->{oracle_pwd} = $options{password}; } elsif (($k eq 'is_mysql') && $options{is_mysql}) { $self->{is_mysql} = $options{is_mysql}; } elsif (($k eq 'is_mssql') && $options{is_mssql}) { $self->{is_mssql} = $options{is_mssql}; } elsif ($k eq 'where') { while ($options{where} =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) { my $table = $1; my $where = $2; $where =~ s/^\s+//; $where =~ s/\s+$//; $self->{where}{$table} = $where; } if ($options{where}) { $self->{global_where} = $options{where}; } } elsif ($k eq 'drop_if_exists') # do not override config parameter, when option is not set { $self->{$k} ||= $options{$k}; } elsif ($options{$k} ne '') { $self->{"\L$k\E"} = $options{$k}; } } # Do not allow global allow/exclude with SHOW_* reports if ($self->{type} eq 'SHOW_REPORT' && ($#{$self->{limited}{ALL}} >= 0 || $#{$self->{excluded}{ALL}} >= 0)) { $self->logit("FATAL: you can not use global filters in ALLOW/EXCLUDE directive with SHOW_REPORT\n", 0, 1); } # Global regex will be applied to the export type only foreach my $i (@{$self->{limited}{ALL}}) { my $typ = $self->{type} || 'TABLE'; $typ = 'TABLE' if ($self->{type} =~ /(SHOW_TABLE|SHOW_COLUMN|FDW|KETTLE|COPY|INSERT|TEST)/); push(@{$self->{limited}{$typ}}, $i); } delete $self->{limited}{ALL}; foreach my $i (@{$self->{excluded}{ALL}}) { my $typ = $self->{type} || 'TABLE'; $typ = 'TABLE' if ($self->{type} =~ /(SHOW_TABLE|SHOW_COLUMN|FDW|KETTLE|COPY|INSERT|TEST)/); push(@{$self->{excluded}{$typ}}, $i); } delete $self->{excluded}{ALL}; $self->{debug} = $AConfig{'DEBUG'} if ($AConfig{'DEBUG'} >= 1); # Set default XML data extract method if (not defined $self->{xml_pretty} || ($self->{xml_pretty} != 0)) { $self->{xml_pretty} = 1; } # Set a default name for the foreign server if (!$self->{fdw_server} && $self->{type} eq 'FDW') { $self->{fdw_server} = 'orcl'; } # Validate the oracle_fdw copy mode and format, and set a default mode and format if undefined if ($self->{fdw_server} && $self->{type} eq 'COPY') { $self->{oracle_fdw_copy_mode} ||= 'local'; $self->{oracle_fdw_copy_mode} = lc($self->{oracle_fdw_copy_mode}); if (!grep(/^$self->{oracle_fdw_copy_mode}$/, @ORACLE_FDW_COPY_MODES)) { $self->logit("FATAL: Unknown oracle_fdw copy mode: $self->{oracle_fdw_copy_mode}. Valid modes: " . join(', ', @ORACLE_FDW_COPY_MODES) . "\n",0,1); } $self->{oracle_fdw_copy_format} ||= 'binary'; $self->{oracle_fdw_copy_format} = lc($self->{oracle_fdw_copy_format}); if (!grep(/^$self->{oracle_fdw_copy_format}$/, @ORACLE_FDW_COPY_FORMATS)) { $self->logit("FATAL: Unknown oracle_fdw copy format: $self->{oracle_fdw_copy_format}. Valid formats: " . join(', ', @ORACLE_FDW_COPY_FORMATS) . "\n",0,1); } } # Set the schema where the foreign tables will be created $self->{fdw_import_schema} ||= 'ora2pg_fdw_import'; # By default we will drop the temporary schema used for foreign import if (not defined $self->{drop_foreign_schema} || $self->{drop_foreign_schema} != 0) { $self->{drop_foreign_schema} = 1; } # By default varchar without size constraint are translated into text if (not defined $self->{varchar_to_text} || $self->{varchar_to_text} != 0) { $self->{varchar_to_text} = 1; } # Should we use \i or \ir in psql scripts if ($AConfig{PSQL_RELATIVE_PATH}) { $self->{psql_relative_path} = 'r'; } else { $self->{psql_relative_path} = ''; } # Clean potential remaining temporary files my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); unlink($dirprefix . 'temp_pass2_file.dat'); unlink($dirprefix . 'temp_cost_file.dat'); # Autodetexct if we are exporting a MySQL database if ($self->{oracle_dsn} =~ /dbi:mysql/i) { $self->{is_mysql} = 1; } elsif ($self->{oracle_dsn} =~ /dbi:ODBC:driver=msodbcsql/i) { $self->{is_mssql} = 1; } # Preload our dedicated function per DBMS if ($self->{is_mysql}) { @{$self->{sysusers}} = (); import Ora2Pg::MySQL; $self->{sgbd_name} = 'MySQL'; } elsif ($self->{is_mssql}) { push(@{$self->{sysusers}}, 'sys'); import Ora2Pg::MSSQL; $self->{sgbd_name} = 'MSSQL'; } else { import Ora2Pg::Oracle; $self->{sgbd_name} = 'Oracle'; } # Export json configuration test $self->{json_test} ||= 0; # Show dependencies between stored procedures $self->{print_dependencies} ||= 0; # Set default system user/schema to not export. Most of them are extracted from this doc: # http://docs.oracle.com/cd/E11882_01/server.112/e10575/tdpsg_user_accounts.htm#TDPSG20030 if (!$self->{is_mysql} && !$self->{is_mssql}) { push(@{$self->{sysusers}},'SYSTEM','CTXSYS','DBSNMP','EXFSYS','LBACSYS','MDSYS','MGMT_VIEW','OLAPSYS','ORDDATA','OWBSYS','ORDPLUGINS','ORDSYS','OUTLN','SI_INFORMTN_SCHEMA','SYS','SYSMAN','WK_TEST','WKSYS','WKPROXY','WMSYS','XDB','APEX_PUBLIC_USER','DIP','FLOWS_020100','FLOWS_030000','FLOWS_040100','FLOWS_010600','FLOWS_FILES','MDDATA','ORACLE_OCM','SPATIAL_CSW_ADMIN_USR','SPATIAL_WFS_ADMIN_USR','XS$NULL','PERFSTAT','SQLTXPLAIN','DMSYS','TSMSYS','WKSYS','APEX_040000','APEX_040200','DVSYS','OJVMSYS','GSMADMIN_INTERNAL','APPQOSSYS','DVSYS','DVF','AUDSYS','APEX_030200','MGMT_VIEW','ODM','ODM_MTR','TRACESRV','MTMSYS','OWBSYS_AUDIT','WEBSYS','WK_PROXY','OSE$HTTP$ADMIN','AURORA$JIS$UTILITY$','AURORA$ORB$UNAUTHENTICATED','DBMS_PRIVILEGE_CAPTURE','CSMIG', 'MGDSYS', 'SDE','DBSFWUSER'); } # Log file handle $self->{fhlog} = undef; if ($self->{logfile}) { $self->{fhlog} = new IO::File; $self->{fhlog}->open(">>$self->{logfile}") or $self->logit("FATAL: can't log to $self->{logfile}, $!\n", 0, 1); } # Autoconvert SRID if (not defined $self->{convert_srid} || ($self->{convert_srid} != 0)) { $self->{convert_srid} = 1; } if (not defined $self->{default_srid}) { $self->{default_srid} = 4326; } # Default function to use for ST_Geometry $self->{st_srid_function} ||= 'ST_SRID'; $self->{st_dimension_function} ||= 'ST_DIMENSION'; $self->{st_geometrytype_function} ||= 'ST_GeometryType'; $self->{st_asbinary_function} ||= 'ST_AsBinary'; $self->{st_astext_function} ||= 'ST_AsText'; # Force Ora2Pg to extract spatial object in binary format $self->{geometry_extract_type} = uc($self->{geometry_extract_type}); if (!$self->{geometry_extract_type} || !grep(/^$self->{geometry_extract_type}$/, 'WKT','WKB','INTERNAL')) { $self->{geometry_extract_type} = 'INTERNAL'; } # Default value for triming can be LEADING, TRAILING or BOTH $self->{trim_type} = 'BOTH' if (!$self->{trim_type} || !grep(/^$self->{trim_type}/, 'BOTH', 'LEADING', 'TRAILING')); # Default triming character is space $self->{trim_char} = ' ' if ($self->{trim_char} eq ''); # Disable the use of orafce library by default $self->{use_orafce} ||= 0; if ($self->{type} eq 'SHOW_REPORT') { $self->{use_orafce} = 0; } # Disable the use of mssqlfce library by default $self->{use_mssqlfce} ||= 0; $self->{local_schemas} = (); $self->{local_schemas_regex} = ''; # Do not apply any default table filtering to improve performances by not applying regexp $self->{no_excluded_table} ||= 0; # Enable BLOB data export by default if (not defined $self->{enable_blob_export}) { $self->{enable_blob_export} = 1; } # Enable CLOB data export by default if (not defined $self->{enable_clob_export}) { $self->{enable_clob_export} = 1; } # Table data export will be sorted by name by default $self->{data_export_order} ||= 'name'; $self->{export_gtt} = 0 if ($self->{type} ne 'TABLE'); # Free some memory %options = (); %AConfig = (); # Enable create or replace by default if ($self->{create_or_replace} || not defined $self->{create_or_replace}) { $self->{create_or_replace} = ' OR REPLACE'; } else { $self->{create_or_replace} = ''; } # Export to files using lo_import to be loaded # into PG need blob_to_lo to be activated and # some other configuration disabled. if ($self->{lo_import}) { if ($self->{type} eq 'INSERT') { $self->logit("FATAL: You must use action COPY, not INSERT with --lo_import.\n", 0, 1); } $self->{blob_to_lo} = 1; $self->{truncate_table} = 0; $self->{drop_fkeys} = 0; $self->{drop_indexes} = 0; } $self->{copy_freeze} = ' FREEZE' if ($self->{copy_freeze}); # Prevent use of COPY FREEZE with some incompatible case if ($self->{copy_freeze}) { if ($self->{pg_dsn} && ($self->{jobs} > 1)) { $self->logit("FATAL: You can not use COPY FREEZE with -j (JOBS) > 1 and direct import to PostgreSQL.\n", 0, 1); } elsif ($self->{oracle_copies} > 1) { $self->logit("FATAL: You can not use COPY FREEZE with -J (ORACLE_COPIES) > 1.\n", 0, 1); } } else { $self->{copy_freeze} = ''; } # Multiprocess init $self->{jobs} ||= 1; $self->{child_count} = 0; # backward compatibility if ($self->{thread_count}) { $self->{jobs} = $self->{thread_count} || 1; } $self->{has_utf8_fct} = 1; eval { utf8::valid("test utf8 function"); }; if ($@) { # Old perl install doesn't include these functions $self->{has_utf8_fct} = 0; } if ($self->{is_mysql} or $self->{is_mssql}) { # MySQL and MSQL do not supports this syntax fallback to read committed $self->{transaction} =~ s/(READ ONLY|READ WRITE)/ISOLATION LEVEL READ COMMITTED/; } # Set Oracle, Perl and PostgreSQL encoding that will be used $self->_init_environment(); # Backward compatibility $self->{rename_partition} = 1 if (!$self->{rename_partition} && $self->{prefix_partition}); # Multiple Oracle connection $self->{oracle_copies} ||= 0; $self->{ora_conn_count} = 0; $self->{data_limit} ||= 10000; $self->{blob_limit} ||= 0; $self->{clob_as_blob} ||= 0; $self->{disable_partition} ||= 0; $self->{parallel_tables} ||= 0; $self->{use_lob_locator} ||= 0; $self->{disable_partition} = 1 if ($self->{is_mssql} and ($self->{type} eq 'COPY' or $self->{type} eq 'INSERT')); # Transformation and output during data export $self->{oracle_speed} ||= 0; $self->{ora2pg_speed} ||= 0; if (($self->{oracle_speed} || $self->{ora2pg_speed}) && !grep(/^$self->{type}$/, 'COPY', 'INSERT', 'DATA')) { # No output is only available for data export. die "FATAL: --oracle_speed or --ora2pg_speed can only be use with data export.\n"; } $self->{oracle_speed} = 1 if ($self->{ora2pg_speed}); # Shall we prefix function with a schema name to emulate a package? $self->{package_as_schema} = 1 if (not exists $self->{package_as_schema} || ($self->{package_as_schema} eq '')); $self->{package_functions} = (); # Set user defined data type translation if ($self->{data_type}) { $self->{data_type} =~ s/\\,/#NOSEP#/gs; my @transl = split(/[,;]/, uc($self->{data_type})); $self->{data_type} = (); # Set default type conversion if ($self->{is_mysql}) { %{$self->{data_type}} = %Ora2Pg::MySQL::SQL_TYPE; } elsif ($self->{is_mssql}) { %{$self->{data_type}} = %Ora2Pg::MSSQL::SQL_TYPE; } else { %{$self->{data_type}} = %Ora2Pg::Oracle::SQL_TYPE; } # then set custom type conversion from the DATA_TYPE # configuration directive foreach my $t (@transl) { my ($typ, $val) = split(/:/, $t); $typ =~ s/^\s+//; $typ =~ s/\s+$//; $val =~ s/^\s+//; $val =~ s/\s+$//; $typ =~ s/#NOSEP#/,/g; $val =~ s/#NOSEP#/,/g; $self->{data_type}{$typ} = lc($val) if ($val); } } else { # Set default type conversion if ($self->{is_mysql}) { %{$self->{data_type}} = %Ora2Pg::MySQL::SQL_TYPE; } elsif ($self->{is_mssql}) { %{$self->{data_type}} = %Ora2Pg::MSSQL::SQL_TYPE; } else { %{$self->{data_type}} = %Ora2Pg::Oracle::SQL_TYPE; } } # Set some default $self->{global_where} ||= ''; $self->{global_delete} ||= ''; $self->{prefix} = 'DBA'; if ($self->{user_grants}) { $self->{prefix} = 'ALL'; } $self->{bzip2} ||= '/usr/bin/bzip2'; $self->{default_numeric} ||= 'bigint'; $self->{type_of_type} = (); $self->{dump_as_html} ||= 0; $self->{dump_as_csv} ||= 0; $self->{dump_as_json} ||= 0; $self->{dump_as_sheet} ||= 0; $self->{dump_as_file_prefix} ||= ''; $self->{top_max} ||= 10; $self->{print_header} ||= 0; $self->{use_default_null} = 1 if (!defined $self->{use_default_null}); $self->{estimate_cost} = 1 if ($self->{dump_as_sheet}); $self->{count_rows} ||= 0; $self->{count_rows} = 1 if ($self->{type} eq 'TEST_COUNT'); # Enforce preservation of primary and unique keys # when USE_TABLESPACE is enabled if ($self->{use_tablespace} && !$self->{keep_pkey_names}) { print STDERR "WARNING: Enforcing KEEP_PKEY_NAMES to 1 because USE_TABLESPACE is enabled.\n"; $self->{keep_pkey_names} = 1; } # DATADIFF defaults $self->{datadiff} ||= 0; $self->{datadiff_del_suffix} ||= '_del'; $self->{datadiff_ins_suffix} ||= '_ins'; $self->{datadiff_upd_suffix} ||= '_upd'; # Internal date boundary. Date below will be added to 2000, others will used 1900 $self->{internal_date_max} ||= 49; # Set the target PostgreSQL major version if (!$self->{pg_version}) { print STDERR "WARNING: target PostgreSQL version must be set in PG_VERSION configuration directive. Using default: 11\n"; $self->{pg_version} = 11; } if ($self->{pg_version} >= 15) { $self->{pg_supports_negative_scale} //= 1; } # Compatibility with PostgreSQL versions if ($self->{pg_version} >= 9.0) { $self->{pg_supports_when} //= 1; $self->{pg_supports_ifexists} //= 1; } if ($self->{pg_supports_ifexists} == 1) { $self->{pg_supports_ifexists} = 'IF EXISTS'; } if ($self->{pg_version} >= 9.1) { $self->{pg_supports_insteadof} //= 1; } if ($self->{pg_version} >= 9.3) { $self->{pg_supports_mview} //= 1; $self->{pg_supports_lateral} //= 1; } if ($self->{pg_version} >= 9.4) { $self->{pg_supports_checkoption} //= 1; } if ($self->{pg_version} >= 9.5) { $self->{pg_supports_named_operator} //= 1; } if ($self->{pg_version} >= 10) { $self->{pg_supports_partition} //= 1; $self->{pg_supports_identity} //= 1; } if ($self->{pg_version} >= 11) { $self->{pg_supports_procedure} //= 1; } if ($self->{pg_version} >= 12) { $self->{pg_supports_virtualcol} //= 1; } if ($self->{pg_version} >= 14) { $self->{pg_supports_outparam} //= 1; } if (!$self->{pg_supports_procedure}) { $self->{pg_supports_outparam} = 0; } # Other PostgreSQL fork compatibility # Redshift if ($self->{pg_supports_substr} eq '') { $self->{pg_supports_substr} //= 1; } $self->{pg_background} ||= 0; # Backward compatibility with LongTrunkOk with typo if ($self->{longtrunkok} && not defined $self->{longtruncok}) { $self->{longtruncok} = $self->{longtrunkok}; } $self->{use_lob_locator} = 0 if ($self->{is_mssql}); $self->{longtruncok} = 0 if (not defined $self->{longtruncok}); # With lob locators LONGREADLEN must at least be 1MB if (!$self->{longreadlen} || $self->{use_lob_locator}) { $self->{longreadlen} = (1023*1024); $self->{longtruncok} = 1; } # Limit he number of row extracted from MSSQL $self->{select_top} ||= 0; $self->{select_top} = 0 if (!$self->{is_mssql}); # Backward compatibility with PG_NUMERIC_TYPE alone $self->{pg_integer_type} = 1 if (not defined $self->{pg_integer_type}); # Backward compatibility with CASE_SENSITIVE $self->{preserve_case} = $self->{case_sensitive} if (defined $self->{case_sensitive} && not defined $self->{preserve_case}); $self->{schema} = uc($self->{schema}) if (!$self->{preserve_case} && ($self->{oracle_dsn} !~ /:mysql/i)); # With MySQL override schema with the database name if ($self->{oracle_dsn} =~ /:mysql:.*database=([^;]+)/i) { if ($self->{schema} ne $1) { $self->{schema} = $1; #$self->logit("WARNING: setting SCHEMA to MySQL database name $1.\n", 0); } if (!$self->{schema}) { $self->logit("FATAL: cannot find a valid mysql database in DSN, $self->{oracle_dsn}.\n", 0, 1); } } # Force disabling USE_LOB_LOCATOR with WKT geometry export type, # ST_GeomFromText and SDO_UTIL.TO_WKTGEOMETRY functions return a # CLOB instead of a geometry object if ($self->{use_lob_locator} && uc($self->{geometry_extract_type}) eq 'WKT') { #$self->logit("WARNING: disabling USE_LOB_LOCATOR with WKT geometry export.\n", 0); $self->{use_lob_locator} = 0; } if (($self->{standard_conforming_strings} =~ /^off$/i) || ($self->{standard_conforming_strings} == 0)) { $self->{standard_conforming_strings} = 0; } else { $self->{standard_conforming_strings} = 1; } if (!defined $self->{compile_schema} || $self->{compile_schema}) { $self->{compile_schema} = 1; } else { $self->{compile_schema} = 0; } $self->{export_invalid} ||= 0; $self->{use_reserved_words} ||= 0; $self->{pkey_in_create} ||= 0; $self->{security} = (); # Should we add SET ON_ERROR_STOP to generated SQL files $self->{stop_on_error} = 1 if (not defined $self->{stop_on_error}); # Force foreign keys to be created initialy deferred if export type # is TABLE or to set constraint deferred with data export types/ $self->{defer_fkey} ||= 0; # How to export partition by reference (none, duplicate or number of hash) $self->{partition_by_reference} ||= 'none'; # Allow multiple or chained extraction export type $self->{export_type} = (); if ($self->{type}) { @{$self->{export_type}} = split(/[\s,;]+/, $self->{type}); # Assume backward compatibility with DATA replacement by INSERT map { s/^DATA$/INSERT/; } @{$self->{export_type}}; } else { @{$self->{export_type}} = ('TABLE'); } # If you decide to autorewrite PLSQL code, this load the dedicated # Perl module $self->{plsql_pgsql} = 1 if ($self->{plsql_pgsql} eq ''); $self->{plsql_pgsql} = 1 if ($self->{estimate_cost}); if ($self->{plsql_pgsql}) { use Ora2Pg::PLSQL; } $self->{fhout} = undef; $self->{compress} = ''; $self->{pkgcost} = 0; $self->{total_pkgcost} = 0; if ($^O =~ /MSWin32|dos/i) { if ( ($self->{oracle_copies} > 1) || ($self->{jobs} > 1) || ($self->{parallel_tables} > 1) ) { $self->logit("WARNING: multiprocess is not supported under that kind of OS.\n", 0); $self->logit("If you need full speed at data export, please use Linux instead.\n", 0); } $self->{oracle_copies} = 0; $self->{jobs} = 0; $self->{parallel_tables} = 0; } if ($self->{parallel_tables} > 1) { $self->{file_per_table} = 1; } if ($self->{jobs} > 1) { $self->{file_per_function} = 1; } if ($self->{debug}) { $self->logit("Ora2Pg version: $VERSION\n"); $self->logit("Export type: $self->{type}\n", 1); $self->logit("Geometry export type: $self->{geometry_extract_type}\n", 1); } # Replace ; or space by comma in the audit user list $self->{audit_user} =~ s/[;\s]+/,/g; # TEST* action need PG_DSN to be set if ($self->{type} =~ /^TEST/ && !$self->{pg_dsn}) { $self->logit("FATAL: export type $self->{type} required PG_DSN to be set.\n", 0, 1); } # FOREIGN_SERVER and PARTITION_BY_REFERENCE set to duplicate is not possible if ($self->{partition_by_reference} eq 'duplicate' && $self->{fdw_server}) { $self->logit("FATAL: PARTITION_BY_REFERENCE set to duplicate with FDW_SERVER set is not supported.\n", 0, 1); } # Set the PostgreSQL connection information for data import or to # defined the dblink connection to use in autonomous transaction $self->set_pg_conn_details(); # Set stdout encoding to UTF8 to avoid "Wide character in print" warning if ($^O !~ /MSWin32|dos/i) { binmode(STDOUT, "encoding(UTF-8)"); } # Mark that we are exporting data using oracle_fdw $self->{oracle_fdw_data_export} = 0; if ($self->{fdw_server} && $self->{type} =~ /^(INSERT|COPY)$/) { $self->{oracle_fdw_data_export} = 1; } if (!$self->{input_file}) { if ($self->{type} eq 'LOAD') { $self->logit("FATAL: with LOAD you must provide an input file\n", 0, 1); } if (!$self->{oracle_dsn} || ($self->{oracle_dsn} =~ /;sid=SIDNAME/)) { $self->logit("FATAL: you must set ORACLE_DSN in ora2pg.conf or use a DDL input file.\n", 0, 1); } # Connect the database if ($self->{oracle_dsn} =~ /dbi:mysql/i) { $self->{is_mysql} = 1; } elsif ($self->{oracle_dsn} =~ /dbi:ODBC:driver=msodbcsql/i) { $self->{is_mssql} = 1; } $self->{dbh} = $self->_db_connection(); # Get the Oracle version $self->{db_version} = $self->_get_version(); # Compile again all objects in the schema if (!$self->{is_mysql} && !$self->{is_mssql} && $self->{compile_schema}) { $self->_compile_schema(uc($self->{compile_schema})); } if (!grep(/^$self->{type}$/, 'COPY', 'INSERT', 'SEQUENCE', 'SEQUENCE_VALUES', 'GRANT', 'TABLESPACE', 'QUERY', 'SYNONYM', 'FDW', 'KETTLE', 'DBLINK', 'DIRECTORY') && $self->{type} !~ /SHOW_/) { if ($self->{plsql_pgsql} && !$self->{no_function_metadata}) { my @done = (); if ($#{ $self->{look_forward_function} } >= 0) { foreach my $o (@{ $self->{look_forward_function} }) { next if (grep(/^$o$/i, @done) || uc($o) eq uc($self->{schema})); push(@done, $o); if ($self->{type} eq 'VIEW') { # Limit to package lookup with VIEW export type $self->_get_package_function_list($o) if (!$self->{is_mysql} && !$self->{is_mssql}); } else { # Extract all package/function/procedure meta information $self->_get_plsql_metadata($o); } } } if ($self->{type} eq 'VIEW') { # Limit to package lookup with WIEW export type $self->_get_package_function_list() if (!$self->{is_mysql} && !$self->{is_mssql}); } else { # Extract all package/function/procedure meta information $self->_get_plsql_metadata(); } } $self->{security} = $self->_get_security_definer($self->{type}) if (grep(/^$self->{type}$/, 'TRIGGER', 'FUNCTION','PROCEDURE','PACKAGE')); } } else { $self->{plsql_pgsql} = 1; $self->replace_tables(%{$self->{'replace_tables'}}); $self->replace_cols(%{$self->{'replace_cols'}}); if (grep(/^$self->{type}$/, 'TABLE', 'SEQUENCE', 'SEQUENCE_VALUES', 'GRANT', 'TABLESPACE', 'VIEW', 'TRIGGER', 'QUERY', 'FUNCTION','PROCEDURE','PACKAGE','TYPE','SYNONYM', 'DIRECTORY', 'DBLINK', 'LOAD', 'SCRIPT')) { if ($self->{type} eq 'LOAD') { if (!$self->{pg_dsn}) { $self->logit("FATAL: You must set PG_DSN to connect to PostgreSQL to be able to dispatch load over multiple connections.\n", 0, 1); } elsif ($self->{jobs} <= 1) { $self->logit("FATAL: You must set set -j (JOBS) > 1 to be able to dispatch load over multiple connections.\n", 0, 1); } } $self->export_schema(); } else { $self->logit("FATAL: bad export type using input file option\n", 0, 1); } return; } # Register export structure modification if ($self->{type} =~ /^(INSERT|COPY|TABLE|TEST|TEST_DATA)$/) { for my $t (keys %{$self->{'modify_struct'}}) { $self->modify_struct($t, @{$self->{'modify_struct'}{$t}}); } for my $t (keys %{$self->{'exclude_columns'}}) { $self->exclude_columns($t, @{$self->{'exclude_columns'}{$t}}); } # Look for custom data type if ($self->{is_mssql}) { $self->logit("Looking for user defined data type of type FROM => DOMAIN...\n", 1); $self->_get_types(); } if ($self->{type} eq 'TEST_DATA' && !$self->{schema}) { $self->logit("FATAL: the TEST_DATA action requires the SCHEMA directive to be set.\n", 0, 1); } } if ($self->{oracle_fdw_data_export} && scalar keys %{$self->{'modify_struct'}} > 0) { $self->logit("FATAL: MODIFY_STRUCT is not supported with oracle_fdw data export.\n", 0, 1); } if ($self->{oracle_fdw_data_export} && scalar keys %{$self->{'exclude_columns'}} > 0) { $self->logit("FATAL: EXCLUDE_COLUMNS is not supported with oracle_fdw data export.\n", 0, 1); } # backup output filename in multiple export mode $self->{output_origin} = ''; if ($#{$self->{export_type}} > 0) { $self->{output_origin} = $self->{output}; } # Retreive all export types information foreach my $t (@{$self->{export_type}}) { $self->{type} = $t; if (($self->{type} eq 'TABLE') || ($self->{type} eq 'FDW') || ($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY') || ($self->{type} eq 'KETTLE')) { $self->{plsql_pgsql} = 1; # Partitionned table do not accept NOT VALID constraint if ($self->{pg_supports_partition} && $self->{type} eq 'TABLE') { # Get the list of partition ($self->{partitions}, $self->{partitions_default}) = $self->_get_partitions(); } # Get table informations $self->_tables(); } elsif ($self->{type} eq 'VIEW') { $self->_views(); } elsif ($self->{type} eq 'SYNONYM') { $self->_synonyms(); } elsif ($self->{type} eq 'GRANT') { $self->_grants(); } elsif ($self->{type} eq 'SEQUENCE' || $self->{type} eq 'SEQUENCE_VALUES') { $self->_sequences(); } elsif ($self->{type} eq 'TRIGGER') { $self->_triggers(); } elsif ($self->{type} eq 'FUNCTION') { $self->_functions(); } elsif ($self->{type} eq 'PROCEDURE') { $self->_procedures(); } elsif ($self->{type} eq 'PACKAGE') { $self->_packages(); } elsif ($self->{type} eq 'TYPE') { $self->_types(); } elsif ($self->{type} eq 'TABLESPACE') { # Partitionned table do not accept NOT VALID constraint if ($self->{pg_supports_partition}) { # Get the list of partition ($self->{partitions}, $self->{partitions_default}) = $self->_get_partitions(); ($self->{subpartitions}, $self->{subpartitions_default}) = $self->_get_subpartitions(); } $self->_tablespaces(); } elsif ($self->{type} eq 'PARTITION') { $self->_partitions(); } elsif ($self->{type} eq 'DBLINK') { $self->_dblinks(); } elsif ($self->{type} eq 'DIRECTORY') { $self->_directories(); } elsif ($self->{type} eq 'MVIEW') { $self->_materialized_views(); } elsif ($self->{type} eq 'QUERY') { $self->_queries(); } elsif ( ($self->{type} eq 'SHOW_REPORT') || ($self->{type} eq 'SHOW_VERSION') || ($self->{type} eq 'SHOW_SCHEMA') || ($self->{type} eq 'SHOW_TABLE') || ($self->{type} eq 'SHOW_COLUMN') || ($self->{type} eq 'SHOW_ENCODING')) { $self->_show_infos($self->{type}); $self->{dbh}->disconnect() if ($self->{dbh}); exit 0; } elsif ($self->{type} eq 'TEST') { $self->replace_tables(%{$self->{'replace_tables'}}); $self->replace_cols(%{$self->{'replace_cols'}}); $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); # Check if all tables have the same number of indexes, constraints, etc. $self->_test_table(); # Count each object at both sides foreach my $o ('VIEW', 'MVIEW', 'SEQUENCE', 'TYPE', 'FDW') { next if ($self->{is_mysql} && grep(/^$o$/, 'MVIEW','TYPE','FDW')); next if ($self->{is_mssql} && grep(/^$o$/, 'FDW')); $self->_count_object($o); } # count function/procedure/package function $self->_test_function(); # compare sequences values except for mysql as we don't know how to get # the correspondance with mysql auto_increment and PG sequences if (!$self->{is_mysql}) { $self->_test_seq_values(); } # Count row in each table if ($self->{count_rows}) { $self->_table_row_count(); } $self->{dbhdest}->disconnect() if ($self->{dbhdest}); $self->{dbh}->disconnect() if ($self->{dbh}); exit 0; } elsif ($self->{type} eq 'TEST_COUNT') { $self->replace_tables(%{$self->{'replace_tables'}}); $self->replace_cols(%{$self->{'replace_cols'}}); $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); # Count row in each table $self->_table_row_count(); $self->{dbhdest}->disconnect() if ($self->{dbhdest}); $self->{dbh}->disconnect() if ($self->{dbh}); exit 0; } elsif ($self->{type} eq 'TEST_VIEW') { $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); $self->_unitary_test_views(); $self->{dbhdest}->disconnect() if ($self->{dbhdest}); $self->{dbh}->disconnect() if ($self->{dbh}); exit 0; } elsif ($self->{type} eq 'TEST_DATA') { $self->replace_tables(%{$self->{'replace_tables'}}); $self->replace_cols(%{$self->{'replace_cols'}}); $self->set_where_clause($self->{'global_where'}, %{$self->{'where'}}); # Create a connection to PostgreSQL $self->{dbhdest} = $self->_send_to_pgdb() if (!$self->{dbhdest}); if ($self->{fdw_server}) { # Create the oracle_fdw extension and the foreign server $self->_create_foreign_server(); # Import the foreign tables following ALLOW or EXCLUDE $self->_import_foreign_schema() if ($self->{drop_foreign_schema}); } else { $self->_tables(); } if (!$self->{is_mysql} && !$self->{is_mssql}) { # Check for DBMS EXECUTE privilege $self->{has_dbms_log_execute_privilege} = Ora2Pg::Oracle::_has_dbms_log_execute_privilege($self); } # Check that data are the same. $self->_data_validation(); $self->{dbhdest}->disconnect() if ($self->{dbhdest}); $self->{dbh}->disconnect() if ($self->{dbh}); exit 0; } else { warn "type option must be (TABLE, VIEW, GRANT, SEQUENCE, SEQUENCE_VALUES, TRIGGER, PACKAGE, FUNCTION, PROCEDURE, PARTITION, TYPE, INSERT, COPY, TABLESPACE, SHOW_REPORT, SHOW_VERSION, SHOW_SCHEMA, SHOW_TABLE, SHOW_COLUMN, SHOW_ENCODING, FDW, MVIEW, QUERY, KETTLE, DBLINK, SYNONYM, DIRECTORY, LOAD, TEST, TEST_COUNT, TEST_VIEW, TEST_DATA), unknown $self->{type}\n"; } $self->replace_tables(%{$self->{'replace_tables'}}); $self->replace_cols(%{$self->{'replace_cols'}}); $self->set_where_clause($self->{'global_where'}, %{$self->{'where'}}); $self->set_delete_clause($self->{'global_delete'}, %{$self->{'delete'}}); } if ( ($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY') || ($self->{type} eq 'KETTLE') ) { if ( ($self->{type} eq 'KETTLE') && !$self->{pg_dsn} ) { $self->logit("FATAL: PostgreSQL connection datasource must be defined with KETTLE export.\n", 0, 1); } elsif ($self->{type} ne 'KETTLE') { if ($self->{defer_fkey} && $self->{pg_dsn}) { $self->logit("FATAL: DEFER_FKEY can not be used with direct import to PostgreSQL, check use of DROP_FKEY instead.\n", 0, 1); } if ($self->{datadiff} && $self->{pg_dsn}) { $self->logit("FATAL: DATADIFF can not be used with direct import to PostgreSQL because direct import may load data in several transactions.\n", 0, 1); } if ($self->{datadiff} && !$self->{pg_supports_lateral}) { $self->logit("FATAL: DATADIFF requires LATERAL support (Pg version 9.3 and above; see config parameter PG_SUPPORTS_LATERAL)\n", 0, 1); } $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); # In case we will use oracle_fdw creates the foreign tables if ($self->{fdw_server} && $self->{pg_dsn}) { # Create the oracle_fdw extension en the foreign server $self->_create_foreign_server(); } } } # Disconnect from the database $self->{dbh}->disconnect() if ($self->{dbh}); } sub _select_output_file_suffix { my ($self, $extension) = @_; # If an output file template is defined if ($self->{dump_as_file_prefix}) { $self->{fhlog} = undef; $self->{fhlog} = new IO::File; $self->{fhlog}->open(">>$self->{dump_as_file_prefix}.$extension") or $self->logit("FATAL: can't log to $self->{dump_as_file_prefix}.$extension, $!\n", 0, 1); if ($self->{debug}){ print STDERR "Saving report to $self->{dump_as_file_prefix}.$extension\n"; } } } # use to set encoding sub _init_environment { my ($self) = @_; # Set default Oracle client encoding if (!$self->{nls_lang}) { if ($self->{is_mysql}) { $self->{nls_lang} = 'utf8'; } elsif ($self->{is_mssql}) { $self->{nls_lang} = 'iso_1'; $self->{client_encoding} = 'LATIN1' if (!$self->{client_encoding}); } else { $self->{nls_lang} = 'AMERICAN_AMERICA.AL32UTF8'; } } if (!$self->{nls_nchar}) { if ($self->{is_mysql}) { $self->{nls_nchar} = 'utf8_general_ci'; } elsif ($self->{is_mssql}) { $self->{nls_nchar} = 'SQL_Latin1_General_CP1_CI_AS'; } else { $self->{nls_nchar} = 'AL32UTF8'; } } $ENV{NLS_LANG} = $self->{nls_lang}; $ENV{NLS_NCHAR} = $self->{nls_nchar}; # Force Perl to use utf8 I/O encoding by default or the # encoding given in the BINMODE configuration directive. # See http://perldoc.perl.org/5.14.2/open.html for values # that can be used. Default is :utf8 $self->set_binmode(); # Set default PostgreSQL client encoding to UTF8 if (!$self->{client_encoding} || $self->{nls_lang} =~ /UTF8/i) { $self->{client_encoding} = 'UTF8'; } } sub set_binmode { my $self = shift; my ($package, $filename, $line) = caller; if ( !$self->{input_file} && (!$self->{'binmode'} || $self->{nls_lang} =~ /UTF8/i) ) { use open ':utf8'; } elsif ($self->{'binmode'} =~ /^:/) { eval "use open '$self->{'binmode'}';"; die "FATAL: can't use open layer $self->{'binmode'}\n" if ($@); } elsif ($self->{'binmode'} and $self->{'binmode'} ne 'raw' and $self->{'binmode'} ne 'locale') { eval "use open ':encoding($self->{'binmode'})';"; die "FATAL: can't use open layer :encoding($self->{'binmode'})\n" if ($@); } # Set default PostgreSQL client encoding to UTF8 if (!$self->{client_encoding} || ($self->{nls_lang} =~ /UTF8/ && !$self->{input_file}) ) { $self->{client_encoding} = 'UTF8'; } if ($#_ == 0) { my $enc = $self->{'binmode'} || 'utf8'; $enc =~ s/^://; if ($self->{'binmode'} eq 'raw' or $self->{'binmode'} eq 'locale') { binmode($_[0], ":$enc"); } else { binmode($_[0], ":encoding($enc)"); } } } sub _is_utf8_file { my $file = shift(); my $utf8 = 0; if (open(my $f, '<', $file)) { local $/; my $data = <$f>; close($f); if (utf8::decode($data)) { $utf8 = 1 } } return $utf8; } # We provide a DESTROY method so that the autoloader doesn't # bother trying to find it. We also close the DB connexion sub DESTROY { my $self = shift; #$self->{dbh}->disconnect() if ($self->{dbh}); } sub set_pg_conn_details { my $self = shift; # Init connection details with configuration options $self->{pg_dsn} ||= ''; $self->{pg_dsn} =~ /dbname=([^;]*)/; $self->{dbname} = $1 || 'testdb'; $self->{pg_dsn} =~ /host=([^;]*)/; $self->{dbhost} = $1 || 'localhost'; $self->{pg_dsn} =~ /port=([^;]*)/; $self->{dbport} = $1 || 5432; $self->{dbuser} = $self->{pg_user} || 'pguser'; $self->{dbpwd} = $self->{pg_pwd} || 'pgpwd'; if (!$self->{dblink_conn}) { #$self->{dblink_conn} = "port=$self->{dbport} dbname=$self->{dbname} host=$self->{dbhost} user=$self->{dbuser} password=$self->{dbpwd}"; # Use a more generic connection string, the password must be # set in .pgpass. Default is to use unix socket to connect. $self->{dblink_conn} = "format('port=%s dbname=%s user=%s', current_setting('port'), current_database(), current_user)"; } } =head2 _send_to_pgdb Open a DB handle to a PostgreSQL database =cut sub _send_to_pgdb { my ($self) = @_; eval("use DBD::Pg qw(:pg_types);"); return if ($self->{oracle_speed}); if (!defined $self->{pg_pwd}) { eval("use Term::ReadKey;"); if (!$@) { $self->{pg_user} = $self->_ask_username('PostgreSQL') unless (defined($self->{pg_user})); $self->{pg_pwd} = $self->_ask_password('PostgreSQL'); } } # Read the password from file each time if the file exists. if (-e $self->{pg_pwd}) { open(FH, '<', $self->{pg_pwd}) or $self->logit("FATAL: can't read PG password file: $self->{pg_pwd, $!}\n", 0, 1); $self->{pg_pwd} = ; chomp($self->{pg_pwd}); close(FH); } $ENV{PGAPPNAME} = 'ora2pg ' || $VERSION; # Connect the destination database my $dbhdest = DBI->connect($self->{pg_dsn}, $self->{pg_user}, $self->{pg_pwd}, {AutoInactiveDestroy => 1, PrintError => 0}); # Check for connection failure if (!$dbhdest) { $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); } # Force execution of initial command $self->_pg_initial_command($dbhdest); return $dbhdest; } =head2 _grants This function is used to retrieve all privilege information. It extracts all Oracle's ROLES to convert them to Postgres groups (or roles) and searches all users associated to these roles. =cut sub _grants { my ($self) = @_; $self->logit("Retrieving users/roles/grants information...\n", 1); ($self->{grants}, $self->{roles}) = $self->_get_privilege(); } =head2 _sequences This function is used to retrieve all sequences information. =cut sub _sequences { my ($self) = @_; $self->logit("Retrieving sequences information...\n", 1); $self->{sequences} = $self->_get_sequences(); } =head2 _triggers This function is used to retrieve all triggers information. =cut sub _triggers { my ($self) = @_; $self->logit("Retrieving triggers information...\n", 1); $self->{triggers} = $self->_get_triggers(); } =head2 _functions This function is used to retrieve all functions information. =cut sub _functions { my $self = shift; $self->logit("Retrieving functions information...\n", 1); $self->{functions} = $self->_get_functions(); } sub start_function_json_config { my ($self, $type) = @_; return if (!$self->{json_test}); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); unlink("${dirprefix}$type.json"); $self->{oracle_dsn} =~ /host=([^;]+)/; my $ora_host = $1 || 'localhost'; $self->{oracle_dsn} =~ /port=(\d+)/; my $ora_port = $1 || ((!$self->{is_mysql}) ? 1521 : 3306); my $sid = ''; if (!$self->{is_mysql}) { $self->{oracle_dsn} =~ /(service_name|sid)=([^;]+)/; $sid = $2 || ''; } else { $self->{oracle_dsn} =~ /(database)=([^;]+)/; $sid = $2 || ''; } my $pg_host = 'localhost'; if ($self->{pg_dsn} =~ /host=([^;]+)/) { $pg_host = $1; } my $pg_port = 5432; if ($self->{pg_dsn} =~ /port=(\d+)/) { $pg_port = $1; } my $pg_db = ''; if ($self->{pg_dsn} =~ /dbname=([^;]+)/) { $pg_db = $1; } my $tfh = $self->append_export_file($dirprefix . "$type.json", 1); flock($tfh, 2) || die "FATAL: can't lock file ${dirprefix}$type.json\n"; $tfh->print(qq/{ "oraConfig": { "dsn": "$self->{oracle_dsn}", "host": "$ora_host", "port": $ora_port, "user": "$self->{oracle_user}", "password": "$self->{oracle_pwd}", "service_name": "$sid", "schema": "$self->{schema}" }, "pgConfig": { "dsn": "$self->{pg_dsn}", "host": "$pg_host", "port": $pg_port, "user": "$self->{pg_user}", "password": "$self->{pg_pwd}", "dbname": "$pg_db", "schema": "$self->{pg_schema}" }, "procfuncConfig": [ /); $self->close_export_file($tfh, 1); } sub end_function_json_config { my ($self, $type) = @_; return if (!$self->{json_test}); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $tfh = $self->append_export_file($dirprefix . "$type.json", 1); flock($tfh, 2) || die "FATAL: can't lock file ${dirprefix}$type.json\n"; # Add an empty json entry at end $tfh->print(qq/ { "routine_type": "", "ora": { "routine_name": "", "return_type": "", "args_list": [ { "0": [ { "name": "", "mode": "", "type": "", "default": "", "value": "" } ] } ] }, "pg": { "routine_name": "", "return_type": "", "args_list": [ { "0": [ { "name": "", "mode": "", "type": "", "default": "", "value": "" } ] } ] } } /); # terminate the json document $tfh->print(qq/ ] } /); $self->close_export_file($tfh, 1); } =head2 _procedures This function is used to retrieve all procedures information. =cut sub _procedures { my $self = shift; $self->logit("Retrieving procedures information...\n", 1); $self->{procedures} = $self->_get_procedures(); } =head2 _packages This function is used to retrieve all packages information. =cut sub _packages { my ($self) = @_; if ($self->{is_mysql} or $self->{is_mssql}) { $self->logit("Action type PACKAGES is not available for $self->{sgbd_name}.\n", 0, 1); } $self->logit("Retrieving packages information...\n", 1); $self->{packages} = $self->_get_packages(); } =head2 _types This function is used to retrieve all custom types information. =cut sub _types { my ($self) = @_; $self->logit("Retrieving user defined types information...\n", 1); $self->{types} = $self->_get_types(); } =head2 _tables This function is used to retrieve all table information. Sets the main hash of the database structure $self->{tables}. Keys are the names of all tables retrieved from the current database. Each table information is composed of an array associated to the table_info key as array reference. In other way: $self->{tables}{$class_name}{table_info} = [(OWNER,TYPE,COMMENT,NUMROW)]; DBI TYPE can be TABLE, VIEW, SYSTEM TABLE, GLOBAL TEMPORARY, LOCAL TEMPORARY, ALIAS, SYNONYM or a data source specific type identifier. This only extracts the TABLE type. It also calls these other private subroutines to affect the main hash of the database structure : @{$self->{tables}{$class_name}{column_info}} = $self->_column_info($class_name, $owner, 'TABLE'); %{$self->{tables}{$class_name}{unique_key}} = $self->_unique_key($class_name, $owner); @{$self->{tables}{$class_name}{foreign_key}} = $self->_foreign_key($class_name, $owner); %{$self->{tables}{$class_name}{check_constraint}} = $self->_check_constraint($class_name, $owner); =cut sub sort_view_by_iter { if (exists $ordered_views{$a}{iter} || exists $ordered_views{$b}{iter}) { return $ordered_views{$a}{iter} <=> $ordered_views{$b}{iter}; } else { return $a cmp $b; } } sub _tables { my ($self, $nodetail) = @_; if ($self->{is_mssql} && $self->{type} eq 'TABLE') { $self->logit("Retrieving table partitioning information...\n", 1); %{ $self->{partitions_list} } = $self->_get_partitioned_table(); } # Get all tables information specified by the DBI method table_info $self->logit("Retrieving table information...\n", 1); # Retrieve tables informations my %tables_infos = $self->_table_info($self->{count_rows}); # Retrieve column identity information if ($self->{type} ne 'FDW') { %{ $self->{identity_info} } = $self->_get_identities(); } if (scalar keys %tables_infos > 0) { if ( grep(/^$self->{type}$/, 'TABLE','SHOW_REPORT','COPY','INSERT') && !$self->{skip_indices} && !$self->{skip_indexes}) { $self->logit("Retrieving index information...\n", 1); my $autogen = 0; $autogen = 1 if (grep(/^$self->{type}$/, 'COPY','INSERT')); my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}, $autogen); foreach my $tb (keys %{$indexes}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; } foreach my $tb (keys %{$idx_type}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; } foreach my $tb (keys %{$idx_tbsp}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{idx_tbsp}} = %{$idx_tbsp->{$tb}}; } foreach my $tb (keys %{$uniqueness}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{uniqueness}} = %{$uniqueness->{$tb}}; } } # Get detailed informations on each tables if (!$nodetail) { $self->logit("Retrieving columns information...\n", 1); # Retrieve all column's details my %columns_infos = $self->_column_info('',$self->{schema}, 'TABLE'); foreach my $tb (keys %columns_infos) { next if (!exists $tables_infos{$tb}); foreach my $c (keys %{$columns_infos{$tb}}) { push(@{$self->{tables}{$tb}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); } } %columns_infos = (); # Retrieve comment of each columns and FK information if not foreign table export if ($self->{type} ne 'FDW' and (!$self->{oracle_fdw_data_export} || $self->{drop_fkey} || $self->{drop_indexes})) { if ($self->{type} eq 'TABLE') { $self->logit("Retrieving comments information...\n", 1); my %columns_comments = $self->_column_comments(); foreach my $tb (keys %columns_comments) { next if (!exists $tables_infos{$tb}); foreach my $c (keys %{$columns_comments{$tb}}) { $self->{tables}{$tb}{column_comments}{$c} = $columns_comments{$tb}{$c}; } } } # Extract foreign keys informations if (!$self->{skip_fkeys}) { $self->logit("Retrieving foreign keys information...\n", 1); my ($foreign_link, $foreign_key) = $self->_foreign_key('',$self->{schema}); foreach my $tb (keys %{$foreign_link}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{foreign_link}} = %{$foreign_link->{$tb}}; } foreach my $tb (keys %{$foreign_key}) { next if (!exists $tables_infos{$tb}); push(@{$self->{tables}{$tb}{foreign_key}}, @{$foreign_key->{$tb}}); } } } } # Retrieve unique keys and check constraint information if not FDW export if ($self->{type} ne 'FDW' and !$self->{oracle_fdw_data_export}) { $self->logit("Retrieving unique keys information...\n", 1); my %unique_keys = $self->_unique_key('',$self->{schema}); foreach my $tb (keys %unique_keys) { next if (!exists $tables_infos{$tb}); foreach my $c (keys %{$unique_keys{$tb}}) { $self->{tables}{$tb}{unique_key}{$c} = $unique_keys{$tb}{$c}; } } %unique_keys = (); if (!$self->{skip_checks}) { $self->logit("Retrieving check constraints information...\n", 1); my %check_constraints = $self->_check_constraint('',$self->{schema}); foreach my $tb (keys %check_constraints) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{check_constraint}} = ( %{$check_constraints{$tb}}); } } } } my @done = (); my $id = 0; # Set the table information for each class found my $i = 1; my $num_total_table = scalar keys %tables_infos; my $count_table = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_table); foreach my $t (sort keys %tables_infos) { if (!$self->{quiet} && !$self->{debug} && ($count_table % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_table, 25, '=', 'tables', "scanning table $t" ), "\r"; } $count_table++; if (grep(/^\Q$t\E$/, @done)) { $self->logit("Duplicate entry found: $t\n", 1); } else { push(@done, $t); } $self->logit("[$i] Scanning table $t ($tables_infos{$t}{num_rows} rows)...\n", 1); # Check of uniqueness of the table if (exists $self->{tables}{$t}{field_name}) { $self->logit("Warning duplicate table $t, maybe a SYNONYM ? Skipped.\n", 1); next; } # Try to respect order specified in the TABLES limited extraction array if ($#{$self->{limited}{TABLE}} > 0) { $self->{tables}{$t}{internal_id} = 0; for (my $j = 0; $j <= $#{$self->{limited}{TABLE}}; $j++) { if (uc($self->{limited}{TABLE}->[$j]) eq uc($t)) { $self->{tables}{$t}{internal_id} = $j; last; } } } # usually TYPE,COMMENT,NUMROW,... $self->{tables}{$t}{table_info}{type} = $tables_infos{$t}{type}; $self->{tables}{$t}{table_info}{comment} = $tables_infos{$t}{comment}; $self->{tables}{$t}{table_info}{num_rows} = $tables_infos{$t}{num_rows}; $self->{tables}{$t}{table_info}{owner} = $tables_infos{$t}{owner}; $self->{tables}{$t}{table_info}{tablespace} = $tables_infos{$t}{tablespace}; $self->{tables}{$t}{table_info}{nested} = $tables_infos{$t}{nested}; $self->{tables}{$t}{table_info}{size} = $tables_infos{$t}{size}; $self->{tables}{$t}{table_info}{auto_increment} = $tables_infos{$t}{auto_increment}; $self->{tables}{$t}{table_info}{connection} = $tables_infos{$t}{connection}; $self->{tables}{$t}{table_info}{nologging} = $tables_infos{$t}{nologging}; $self->{tables}{$t}{table_info}{partitioned} = $tables_infos{$t}{partitioned}; $self->{tables}{$t}{table_info}{temporary} = $tables_infos{$t}{temporary}; $self->{tables}{$t}{table_info}{duration} = $tables_infos{$t}{duration}; $self->{tables}{$t}{table_info}{index_type} = $tables_infos{$t}{index_type}; if (exists $tables_infos{$t}{fillfactor}) { $self->{tables}{$t}{table_info}{fillfactor} = $tables_infos{$t}{fillfactor}; } # Set the fields information if ($self->{type} ne 'SHOW_REPORT') { my $tmp_tbname = $t; if ($self->{is_mysql}) { if ( $t !~ /\./ && $tables_infos{$t}{owner}) { $tmp_tbname = "\`$tables_infos{$t}{owner}\`.\`$t\`"; } else { # in case we already have the schema name, add doublequote $tmp_tbname =~ s/\./\`.\`/; $tmp_tbname = "\`$tmp_tbname\`"; } } elsif ($self->{is_mssql}) { if ( $t !~ /\./ && $tables_infos{$t}{owner}) { $tmp_tbname = "[$tables_infos{$t}{owner}].[$t]"; } else { # in case we already have the schema name, add doublequote $tmp_tbname =~ s/\./\].\[/; $tmp_tbname = "[$tmp_tbname]"; } } else { if ( $t !~ /\./ ) { $tmp_tbname = "\"$tables_infos{$t}{owner}\".\"$t\""; } else { # in case we already have the schema name, add doublequote $tmp_tbname =~ s/\./"."/; $tmp_tbname = "\"$tmp_tbname\""; } } foreach my $k (sort {$self->{tables}{$t}{column_info}{$a}[11] <=> $self->{tables}{$t}{column_info}{$b}[11]} keys %{$self->{tables}{$t}{column_info}}) { $self->{tables}{$t}{type} = 'table'; push(@{$self->{tables}{$t}{field_name}}, $self->{tables}{$t}{column_info}{$k}[0]); push(@{$self->{tables}{$t}{field_type}}, $self->{tables}{$t}{column_info}{$k}[1]); } } $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_table, 25, '=', 'tables', 'end of scanning.'), "\n"; } #### # Get views definition if it must be exported as table #### if ($#{$self->{view_as_table}} >= 0) { my %view_infos = $self->_get_views(); my @exanped_views = (); foreach my $view (sort keys %view_infos) { foreach my $pattern (@{$self->{view_as_table}}) { push(@exanped_views, $view) if ($view =~ /^$pattern$/i); } } # Retrieve comment of each columns my %columns_comments = $self->_column_comments(); foreach my $view (keys %columns_comments) { next if (!exists $view_infos{$view}); next if (!grep(/^$view$/i, @exanped_views)); foreach my $c (keys %{$columns_comments{$view}}) { $self->{tables}{$view}{column_comments}{$c} = $columns_comments{$view}{$c}; } } foreach my $view (sort keys %view_infos) { # Set the table information for each class found # Jump to desired extraction next if (!grep(/^$view$/i, @exanped_views)); $self->logit("Scanning view $view to export as table...\n", 0); $self->{tables}{$view}{type} = 'view'; $self->{tables}{$view}{text} = $view_infos{$view}{text}; $self->{tables}{$view}{owner} = $view_infos{$view}{owner}; $self->{tables}{$view}{iter} = $view_infos{$view}{iter} if (exists $view_infos{$view}{iter}); $self->{tables}{$view}{alias}= $view_infos{$view}{alias}; $self->{tables}{$view}{comment} = $view_infos{$view}{comment}; my $realview = $view; $realview =~ s/"//g; if (!$self->{is_mysql}) { if ($realview !~ /\./) { $realview = "\"$self->{tables}{$view}{owner}\".\"$realview\""; } else { $realview =~ s/\./"."/; $realview = "\"$realview\""; } } # Set the fields information my $sth = $self->{dbh}->prepare("SELECT * FROM $realview WHERE 1=0"); if (!defined($sth)) { warn "Can't prepare statement: $DBI::errstr"; next; } $sth->execute; if ($sth->err) { warn "Can't execute statement: $DBI::errstr"; next; } $self->{tables}{$view}{field_name} = $sth->{NAME}; $self->{tables}{$view}{field_type} = $sth->{TYPE}; my %columns_infos = $self->_column_info($view, $self->{schema}, 'VIEW', @exanped_views); foreach my $tb (keys %columns_infos) { next if ($tb ne $view); foreach my $c (keys %{$columns_infos{$tb}}) { push(@{$self->{tables}{$view}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); } } } } #### # Get materialized views definition if it must be exported as table #### if ($#{$self->{mview_as_table}} >= 0) { my %view_infos = $self->_get_materialized_views(); my @exanped_views = (); foreach my $view (sort keys %view_infos) { foreach my $pattern (@{$self->{mview_as_table}}) { push(@exanped_views, $view) if ($view =~ /^$pattern$/i); } } foreach my $view (sort keys %view_infos) { # Set the table information for each class found # Jump to desired extraction next if (!grep(/^$view$/i, @exanped_views)); $self->logit("Scanning materialized view $view to export as table...\n", 0); if (exists $self->{tables}{$view}) { $self->logit("WARNING: cannot export materialized view $view as table, a table with same name already exists...\n", 0); next; } $self->{tables}{$view}{type} = 'mview'; $self->{tables}{$view}{text} = $view_infos{$view}{text}; $self->{tables}{$view}{owner} = $view_infos{$view}{owner}; my $realview = $view; $realview =~ s/"//g; if (!$self->{is_mysql}) { if ($realview !~ /\./) { $realview = "\"$self->{tables}{$view}{owner}\".\"$realview\""; } else { $realview =~ s/\./"."/; $realview = "\"$realview\""; } } # Set the fields information my $sth = $self->{dbh}->prepare("SELECT * FROM $realview WHERE 1=0"); if (!defined($sth)) { warn "Can't prepare statement: $DBI::errstr"; next; } $sth->execute; if ($sth->err) { warn "Can't execute statement: $DBI::errstr"; next; } $self->{tables}{$view}{field_name} = $sth->{NAME}; $self->{tables}{$view}{field_type} = $sth->{TYPE}; my %columns_infos = $self->_column_info($view, $self->{schema}, 'MVIEW', @exanped_views); foreach my $tb (keys %columns_infos) { next if ($tb ne $view); foreach my $c (keys %{$columns_infos{$tb}}) { push(@{$self->{tables}{$view}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); } } } } # Look at external tables if (!$self->{is_mysql} && ($self->{db_version} !~ /Release 8/)) { %{$self->{external_table}} = $self->_get_external_tables(); } if (!$self->{is_mssql} && $self->{type} eq 'TABLE') { $self->logit("Retrieving table partitioning information...\n", 1); %{ $self->{partitions_list} } = $self->_get_partitioned_table(); %{ $self->{subpartitions_list} } = $self->_get_subpartitioned_table(); } } sub _get_plsql_code { my $str = shift(); my $ct = ''; my @parts = split(/\b(BEGIN|DECLARE|END\s*(?!IF|LOOP|CASE|INTO|FROM|,|\))[^;\s]*\s*;)/i, $str); my $code = ''; my $other = ''; my $i = 0; for (; $i <= $#parts; $i++) { $ct++ if ($parts[$i] =~ /\bBEGIN\b/i); $ct-- if ($parts[$i] =~ /\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,|\))[^;\s]*\s*;/i); if ( ($ct ne '') && ($ct == 0) ) { $code .= $parts[$i]; last; } $code .= $parts[$i]; } $i++; for (; $i <= $#parts; $i++) { $other .= $parts[$i]; } return ($code, $other); } sub _parse_constraint { my ($self, $tb_name, $cur_col_name, $c) = @_; if ($c =~ /^([^\s]+)\s+(UNIQUE|PRIMARY KEY)\s*\(([^\)]+)\)/is) { my $tp = 'U'; $tp = 'P' if ($2 eq 'PRIMARY KEY'); $self->{tables}{$tb_name}{unique_key}{$1} = { ( type => $tp, 'generated' => 0, 'index_name' => $1, columns => () ) }; push(@{$self->{tables}{$tb_name}{unique_key}{$1}{columns}}, split(/\s*,\s*/, $3)); } elsif ($c =~ /^([^\s]+)\s+CHECK\s*\((.*)\)/is) { my $name = $1; my $desc = $2; if ($desc =~ /^([a-z_\$0-9]+)\b/i) { $name .= "_$1"; } my %tmp = ($name => $desc); $self->{tables}{$tb_name}{check_constraint}{constraint}{$name}{condition} = $desc; if ($c =~ /NOVALIDATE/is) { $self->{tables}{$tb_name}{check_constraint}{constraint}{$name}{validate} = 'NOT VALIDATED'; } } elsif ($c =~ /^([^\s]+)\s+FOREIGN KEY\s*(\([^\)]+\))?\s*REFERENCES\s*([^\(\s]+)\s*\(([^\)]+)\)/is) { my $c_name = $1; if ($2) { $cur_col_name = $2; } my $f_tb_name = $3; my @col_list = split(/,/, $4); $c_name =~ s/"//g; $f_tb_name =~ s/"//g; $cur_col_name =~ s/[\("\)]//g; map { s/"//g; } @col_list; if (!$self->{export_schema}) { $f_tb_name =~ s/^[^\.]+\.//; map { s/^[^\.]+\.//; } @col_list; } push(@{$self->{tables}{$tb_name}{foreign_link}{"\U$c_name\E"}{local}}, $cur_col_name); push(@{$self->{tables}{$tb_name}{foreign_link}{"\U$c_name\E"}{remote}{$f_tb_name}}, @col_list); my $deferrable = ''; $deferrable = 'DEFERRABLE' if ($c =~ /DEFERRABLE/); my $deferred = ''; $deferred = 'DEFERRED' if ($c =~ /INITIALLY DEFERRED/); my $novalidate = ''; $novalidate = 'NOT VALIDATED' if ($c =~ /NOVALIDATE/); # CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE,VALIDATED push(@{$self->{tables}{$tb_name}{foreign_key}}, [ ($c_name,'','','',$deferrable,$deferred,'',$tb_name,'','',$novalidate) ]); } } sub _remove_text_constant_part { my ($self, $str) = @_; for (my $i = 0; $i <= $#{$self->{alternative_quoting_regexp}}; $i++) { while ($$str =~ s/$self->{alternative_quoting_regexp}[$i]/\?TEXTVALUE$self->{text_values_pos}\?/s) { $self->{text_values}{$self->{text_values_pos}} = '$$' . $1 . '$$'; $self->{text_values_pos}++; } } $$str =~ s/\\'/ORA2PG_ESCAPE1_QUOTE'/gs; while ($$str =~ s/''/ORA2PG_ESCAPE2_QUOTE/gs) {} while ($$str =~ s/('[^']+')/\?TEXTVALUE$self->{text_values_pos}\?/s) { $self->{text_values}{$self->{text_values_pos}} = $1; $self->{text_values_pos}++; } for (my $i = 0; $i <= $#{$self->{string_constant_regexp}}; $i++) { while ($$str =~ s/($self->{string_constant_regexp}[$i])/\?TEXTVALUE$self->{text_values_pos}\?/s) { $self->{text_values}{$self->{text_values_pos}} = $1; $self->{text_values_pos}++; } } } sub _restore_text_constant_part { my ($self, $str) = @_; $$str =~ s/\?TEXTVALUE(\d+)\?/$self->{text_values}{$1}/gs; $$str =~ s/ORA2PG_ESCAPE2_QUOTE/''/gs; $$str =~ s/ORA2PG_ESCAPE1_QUOTE'/\\'/gs; if ($self->{type} eq 'TRIGGER') { $$str =~ s/(\s+)(NEW|OLD)\.'([^']+)'/$1$2\.$3/igs; } } sub _get_ddl_from_file { my $self = shift; # Load file in a single string my $content = $self->read_input_file($self->{input_file}); $content =~ s/CREATE\s+OR\s+REPLACE/CREATE/gs; $content =~ s/CREATE\s+EDITIONABLE/CREATE/gs; $content =~ s/CREATE\s+NONEDITIONABLE/CREATE/gs; if ($self->{is_mysql}) { $content =~ s/CREATE\s+ALGORITHM=[^\s]+/CREATE/gs; $content =~ s/CREATE\s+DEFINER=[^\s]+/CREATE/gs; $content =~ s/SQL SECURITY DEFINER VIEW/VIEW/gs; } return $content; } sub read_schema_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Clear content from comment and text constant for better parsing if (!$self->{no_clean_comment}) { $self->_remove_comments(\$content, 1); $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; } my $tid = 0; my @statements = split(/\s*;\s*/, $content); foreach $content (@statements) { $content .= ';'; # Remove some unwanted and unused keywords from the statements $content =~ s/\s+(PARALLEL|COMPRESS|CLUSTERED|NONCLUSTERED)\b//igs; $content =~ s/\s+WITH\s+CHECK\s+ADD\s+CONSTRAINT\s+/ ADD CONSTRAINT /igs; if ($content =~ s/TRUNCATE TABLE\s+([^\s;]+)([^;]*);//is) { my $tb_name = $1; $tb_name =~ s/"//gs; if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } $self->{tables}{$tb_name}{truncate_table} = 1; } elsif ($content =~ s/CREATE\s+(GLOBAL|PRIVATE)?\s*(TEMPORARY)?\s*TABLE[\s]+([^\s]+)\s+AS\s+([^;]+);//is) { my $tb_name = $3; $tb_name =~ s/"//gs; my $tb_def = $4; $tb_def =~ s/\s+/ /gs; $self->{tables}{$tb_name}{table_info}{type} = 'TEMPORARY ' if ($2); $self->{tables}{$tb_name}{table_info}{type} .= 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; $self->{tables}{$tb_name}{table_as} = $tb_def; } elsif ($content =~ s/CREATE\s+(GLOBAL|PRIVATE)?\s*(TEMPORARY)?\s*TABLE[\s]+([^\s\(]+)\s*([^;]+);//is) { my $tb_name = $3; my $tb_def = $4; my $tb_param = ''; $tb_name =~ s/"//gs; $self->{tables}{$tb_name}{table_info}{type} = 'TEMPORARY ' if ($2); $self->{tables}{$tb_name}{table_info}{type} .= 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; # Remove goldengate suplemental table logging $tb_def =~ s/SUPPLEMENTAL LOG DATA \(.*?\) COLUMNS//is; # For private temporary table extract the ON COMMIT information (18c) if ($tb_def =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION//is) { $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT PRESERVE ROWS'; } elsif ($tb_def =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION//is) { $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT DROP'; } elsif ($self->{tables}{$tb_name}{table_info}{type} eq 'TEMPORARY ') { # Default for PRIVATE TEMPORARY TABLE $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT DROP'; } # Get table embedded comment if ($tb_def =~ s/COMMENT=["']([^"']+)["']//is) { $self->{tables}{$tb_name}{table_info}{comment} = $1; } $tb_def =~ s/^\(//; my %fct_placeholder = (); my $i = 0; while ($tb_def =~ s/(\([^\(\)]*\))/\%\%FCT$i\%\%/is) { $fct_placeholder{$i} = $1; $i++; }; ($tb_def, $tb_param) = split(/\s*\)\s*/, $tb_def); my @column_defs = split(/\s*,\s*/, $tb_def); map { s/^\s+//; s/\s+$//; } @column_defs; my $pos = 0; my $cur_c_name = ''; foreach my $c (@column_defs) { next if (!$c); # Replace temporary substitution while ($c =~ s/\%\%FCT(\d+)\%\%/$fct_placeholder{$1}/is) { delete $fct_placeholder{$1}; } # Mysql unique key embedded definition, transform it to special parsing $c =~ s/^UNIQUE KEY/INDEX UNIQUE/is; # Remove things that are not possible with postgres $c =~ s/(PRIMARY KEY.*)NOT NULL/$1/is; # Rewrite some parts for easiest/generic parsing my $tbn = $tb_name; $tbn =~ s/\./_/gs; $c =~ s/^(PRIMARY KEY|UNIQUE)/CONSTRAINT o2pu_$tbn $1/is; $c =~ s/^(CHECK[^,;]+)DEFERRABLE\s+INITIALLY\s+DEFERRED/$1/is; $c =~ s/^CHECK\b/CONSTRAINT o2pc_$tbn CHECK/is; $c =~ s/^FOREIGN KEY/CONSTRAINT o2pf_$tbn FOREIGN KEY/is; $c =~ s/\(\s+/\(/gs; # register column name between double quote my $i = 0; my %col_name = (); # Get column name while ($c =~ s/("[^"]+")/\%\%COLNAME$i\%\%/s) { $col_name{$i} = $1; $i++; } if ($c =~ s/^\s*([^\s]+)\s*//s) { my $c_name = $1; $c_name =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; $c =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; if (!$self->{preserve_case}) { $c_name =~ s/"//gs; } # Retrieve all columns information if (!grep(/^\Q$c_name\E$/i, 'CONSTRAINT','INDEX')) { $cur_c_name = $c_name; $c_name =~ s/\./_/gs; my $c_default = ''; my $virt_col = 'NO'; $c =~ s/\s+ENABLE//is; if ($c =~ s/\bGENERATED\s+(ALWAYS|BY\s+DEFAULT)\s+(ON\s+NULL\s+)?AS\s+IDENTITY\s*(.*)//is) { $self->{identity_info}{$tb_name}{$c_name}{generation} = $1; my $options = $3; $self->{identity_info}{$tb_name}{$c_name}{options} = $3; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(SCALE|EXTEND|SESSION)_FLAG: .//isg; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/KEEP_VALUE: .//is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(START WITH):/$1/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(INCREMENT BY):/$1/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/MAX_VALUE:/MAXVALUE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/MIN_VALUE:/MINVALUE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CYCLE_FLAG: N/NO CYCLE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/NOCYCLE/NO CYCLE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CYCLE_FLAG: Y/CYCLE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE_SIZE:/CACHE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE_SIZE:/CACHE/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/ORDER_FLAG: .//is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/,//gs; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s$//s; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE\s+0/CACHE 1/is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOORDER//is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOKEEP//is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOSCALE//is; $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOT\s+NULL//is; # Be sure that we don't exceed the bigint max value, # we assume that the increment is always positive if ($self->{identity_info}{$tb_name}{$c_name}{options} =~ /MAXVALUE\s+(\d+)/is) { $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(MAXVALUE)\s+\d+/$1 9223372036854775807/is; } $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s+/ /igs; } elsif ($c =~ s/\b(GENERATED ALWAYS AS|AS)\s+(.*)//is) { $virt_col = 'YES'; $c_default = $2; $c_default =~ s/\s+VIRTUAL//is; } my $c_type = ''; if ($c =~ s/^\s*ENUM\s*(\([^\(\)]+\))\s*//is) { my %tmp = (); $tmp{name} = lc($tb_name . '_' . $c_name . '_t'); $tmp{pos} = 0; $tmp{code} .= "CREATE TYPE " . $self->quote_object_name($tmp{name}) . " AS ENUM ($1);"; push(@{$self->{types}}, \%tmp); $c_type = $tmp{name}; } elsif ($c =~ s/^([^\s\(]+)\s*//s) { $c_type = $1; } elsif ($c_default) { # Try to guess a type the virtual column was declared without one, # but always default to text and always display a warning. if ($c_default =~ /ROUND\s*\(/is) { $c_type = 'numeric'; } elsif ($c_default =~ /TO_DATE\s\(/is) { $c_type = 'timestamp'; } else { $c_type = 'text'; } print STDERR "WARNING: Virtual column $tb_name.$cur_c_name has no data type defined, using $c_type but you need to check that this is the right type.\n"; } else { next; } if (!$self->{preserve_case}) { $c_type =~ s/"//gs; } my $c_length = ''; my $c_scale = ''; if ($c =~ s/^\(([^\)]+)\)\s*//s) { $c_length = $1; if ($c_length =~ s/\s*,\s*(\d+)\s*//s) { $c_scale = $1; } } my $c_nullable = 1; if ($c =~ s/CONSTRAINT\s*([^\s]+)?\s*NOT NULL//is) { $c_nullable = 0; } elsif ($c !~ /IS\s+NOT\s+NULL/is && $c =~ s/\bNOT\s+NULL//is) { $c_nullable = 0; } if (($c =~ s/(UNIQUE|PRIMARY KEY)\s*\(([^\)]+)\)//is) || ($c =~ s/(UNIQUE|PRIMARY KEY)\s*//is)) { $c_name =~ s/\./_/gs; my $pk_name = 'o2pu_' . $c_name; my $cols = $c_name; if ($2) { $cols = $2; } $self->_parse_constraint($tb_name, $c_name, "$pk_name $1 ($cols)"); } elsif ( ($c =~ s/CONSTRAINT\s([^\s]+)\sCHECK\s*\(([^\)]+)\)//is) || ($c =~ s/CHECK\s*\(([^\)]+)\)//is) ) { $c_name =~ s/\./_/gs; my $pk_name = 'o2pc_' . $c_name; my $chk_search = $1; if ($2) { $pk_name = $1; $chk_search = $2; } $chk_search .= $c if ($c eq ')'); $self->_parse_constraint($tb_name, $c_name, "$pk_name CHECK ($chk_search)"); } elsif ($c =~ s/REFERENCES\s+([^\(\s]+)\s*\(([^\)]+)\)//is) { $c_name =~ s/\./_/gs; my $pk_name = 'o2pf_' . $c_name; my $chk_search = $1 . "($2)"; $chk_search =~ s/\s+//gs; $self->_parse_constraint($tb_name, $c_name, "$pk_name FOREIGN KEY ($c_name) REFERENCES $chk_search"); } my $auto_incr = 0; if ($c =~ s/\s*AUTO_INCREMENT\s*//is) { $auto_incr = 1; } # At this stage only the DEFAULT part might be on the string if ($c =~ /\bDEFAULT\s+/is) { if ($c =~ s/\bDEFAULT\s+('[^']+')\s*//is) { $c_default = $1; } elsif ($c =~ s/\bDEFAULT\s+([^\s]+)\s*$//is) { $c_default = $1; } elsif ($c =~ s/\bDEFAULT\s+(.*)$//is) { $c_default = $1; } $c_default =~ s/"//gs; if ($self->{plsql_pgsql}) { $c_default = Ora2Pg::PLSQL::convert_plsql_code($self, $c_default); } } if ($c_type =~ /date|timestamp/i && $c_default =~ /^'0000-/) { if ($self->{replace_zero_date}) { $c_default = $self->{replace_zero_date}; } else { $c_default =~ s/^'0000-\d+-\d+/'1970-01-01/; } if ($c_default =~ /^[\-]*INFINITY$/) { $c_default .= "::$c_type"; } } # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE push(@{$self->{tables}{$tb_name}{column_info}{$c_name}}, ($c_name, $c_type, $c_length, $c_nullable, $c_default, $c_length, $c_scale, $c_length, $tb_name, '', $virt_col, $pos, $auto_incr)); } elsif (uc($c_name) eq 'CONSTRAINT') { $self->_parse_constraint($tb_name, $cur_c_name, $c); } elsif (uc($c_name) eq 'INDEX') { if ($c =~ /^\s*UNIQUE\s+([^\s]+)\s+\(([^\)]+)\)/) { my $idx_name = $1; my @cols = (); push(@cols, split(/\s*,\s*/, $2)); map { s/^"//; s/"$//; } @cols; $self->{tables}{$tb_name}{unique_key}->{$idx_name}{type} = 'U'; $self->{tables}{$tb_name}{unique_key}->{$idx_name}{generated} = 0; $self->{tables}{$tb_name}{unique_key}->{$idx_name}{index_name} = $idx_name; push(@{$self->{tables}{$tb_name}{unique_key}->{$idx_name}{columns}}, @cols); } elsif ($c =~ /^\s*([^\s]+)\s+\(([^\)]+)\)/) { my $idx_name = $1; my @cols = (); push(@cols, split(/\s*,\s*/, $2)); map { s/^"//; s/"$//; } @cols; push(@{$self->{tables}{$tb_name}{indexes}{$idx_name}}, @cols); } } } else { $c =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; } $pos++; } map {s/^/\t/; s/$/,\n/; } @column_defs; # look for storage information if ($tb_param =~ /TABLESPACE[\s]+([^\s]+)/is) { $self->{tables}{$tb_name}{table_info}{tablespace} = $1; $self->{tables}{$tb_name}{table_info}{tablespace} =~ s/"//gs; } if ($tb_param =~ /PCTFREE\s+(\d+)/is) { # We only take care of pctfree upper than the default if ($1 > 10) { # fillfactor must be >= 10 $self->{tables}{$tb_name}{table_info}{fillfactor} = 100 - &Ora2Pg::Oracle::min(90, $1); } } if ($tb_param =~ /\bNOLOGGING\b/is) { $self->{tables}{$tb_name}{table_info}{nologging} = 1; } if ($tb_param =~ /\bGLOBAL\s+TEMPORARY\b/is) { $self->{tables}{$tb_name}{table_info}{temporary} = 'Y'; } if ($tb_param =~ /ORGANIZATION EXTERNAL/is) { if ($tb_param =~ /DEFAULT DIRECTORY ([^\s]+)/is) { $self->{external_table}{$tb_name}{director} = $1; } $self->{external_table}{$tb_name}{delimiter} = ','; if ($tb_param =~ /FIELDS TERMINATED BY '(.)'/is) { $self->{external_table}{$tb_name}{delimiter} = $1; } if ($tb_param =~ /PREPROCESSOR EXECDIR\s*:\s*'([^']+)'/is) { $self->{external_table}{$tb_name}{program} = $1; } if ($tb_param =~ /LOCATION\s*\(\s*'([^']+)'\s*\)/is) { $self->{external_table}{$tb_name}{location} = $1; } } } elsif ($content =~ s/CREATE\s+(UNIQUE|BITMAP)?\s*INDEX\s+([^\s]+)\s+ON\s+([^\s\(]+)\s*\((.*)\)//is) { my $is_unique = $1; my $idx_name = $2; my $tb_name = $3; my $idx_def = $4; $idx_name =~ s/"//gs; $tb_name =~ s/\s+/ /gs; $idx_def =~ s/\s+/ /gs; $idx_def =~ s/\s*nologging//is; $idx_def =~ s/STORAGE\s*\([^\)]+\)\s*//is; $idx_def =~ s/COMPRESS(\s+\d+)?\s*//is; $idx_def =~ s/\)\s*WITH\s*\(.*//is; $idx_def =~ s/\s+ASC\b//is; $idx_def =~ s/^\s+//s; $idx_def =~ s/\s+$//is; # look include information if ($idx_def =~ s/\s*\)\s*(INCLUDE|INCLUDING)\s*\(([^\)]+)//is) { my $include = $2; $include =~ s/\s+//g; push(@{$self->{tables}{$tb_name}{idx_type}{$idx_name}{type_include}}, split(/\s*,\s*/, $include)); } # look for storage information if ($idx_def =~ s/TABLESPACE\s*([^\s]+)\s*//is) { $self->{tables}{$tb_name}{idx_tbsp}{$idx_name} = $1; $self->{tables}{$tb_name}{idx_tbsp}{$idx_name} =~ s/"//gs; } if ($idx_def =~ s/ONLINE\s*//is) { $self->{tables}{$tb_name}{concurrently}{$idx_name} = 1; } if ($idx_def =~ s/INDEXTYPE\s+IS\s+.*SPATIAL_INDEX//is) { $self->{tables}{$tb_name}{spatial}{$idx_name} = 1; $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'SPATIAL INDEX'; $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_name} = 'SPATIAL_INDEX'; } if ($idx_def =~ s/layer_gtype=([^\s,]+)//is) { $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_constraint} = uc($1); } if ($idx_def =~ s/sdo_indx_dims=(\d)//is) { $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_dims} = $1; } if ($is_unique eq 'BITMAP') { $is_unique = ''; $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_name} = 'BITMAP'; } $self->{tables}{$tb_name}{uniqueness}{$idx_name} = $is_unique || ''; $idx_def =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/$1/isg; if ($self->{plsql_pgsql}) { $idx_def = Ora2Pg::PLSQL::convert_plsql_code($self, $idx_def); } push(@{$self->{tables}{$tb_name}{indexes}{$idx_name}}, $idx_def); $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'NORMAL'; if ($idx_def =~ /\(/s) { $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'FUNCTION-BASED'; } if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } } elsif ($content =~ s/ALTER\s+TABLE\s+([^\s]+)\s+ADD\s*\(*\s*(.*)//is) { my $tb_name = $1; my $tb_def = $2; $tb_name =~ s/"//g; # Oracle allow multiple constraints declaration inside a single ALTER TABLE # CONSTRAINT CK_TblMstCustomerDriver_RentalDepositAmount CHECK ((RentalDepositAmount>=(0))); while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+CHECK\s*([\(]+.*?[\)]+)\s*(ENABLE|DISABLE|VALIDATE|NOVALIDATE|DEFERRABLE|INITIALLY|DEFERRED|USING\s+INDEX|\s+)?([^,]*)//is) { my $constname = $1; my $code = $2; my $states = $3; my $tbspace_move = $4; if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } my $validate = ''; $validate = ' NOT VALID' if ( $states =~ /NOVALIDATE/is); push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E CHECK $code$validate"); if ( $tbspace_move =~ /USING\s+INDEX\s+TABLESPACE\s+([^\s]+)/is) { if ($self->{use_tablespace}) { $tbspace_move = "ALTER INDEX $constname SET TABLESPACE " . lc($1); push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); } } elsif ($tbspace_move =~ /USING\s+INDEX\s+([^\s]+)/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); } } while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+FOREIGN\s+KEY\s*(\(.*?\)\s+REFERENCES\s+[^\s]+\s*\(.*?\))\s*([^,\)]+|$)//is) { my $constname = $1; my $other_def = $3; if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E FOREIGN KEY $2"); if ($other_def =~ /(ON\s+DELETE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; } if ($other_def =~ /(ON\s+UPDATE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; } my $validate = ''; $validate = ' NOT VALID' if ( $other_def =~ /NOVALIDATE/is); $self->{tables}{$tb_name}{alter_table}[-1] .= $validate; } # We can just have one primary key constraint if ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+PRIMARY KEY//is) { my $constname = lc($1); $tb_def =~ s/^[^\(]+//; $tb_def =~ s/\);$//s; if ( $tb_def =~ s/USING\s+INDEX\s+TABLESPACE\s+([^\s]+).*//s) { $tb_def =~ s/\s+$//; if ($self->{use_tablespace}) { my $tbspace_move = "ALTER INDEX $constname SET TABLESPACE $1"; push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); } push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT $constname PRIMARY KEY " . lc($tb_def)); } elsif ($tb_def =~ s/USING\s+INDEX\s+([^\s]+).*//s) { push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD PRIMARY KEY " . lc($tb_def)); $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); } elsif ($tb_def) { push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT $constname PRIMARY KEY " . lc($tb_def)); } if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } } } elsif ($content =~ s/ALTER\s+TABLE\s+([^\s]+)\s+ADD\s+(CONSTRAINT\s+[^\s]+\s+.*)//is) { my $tb_name = $1; my $tb_def = $2; $tb_name =~ s/"//g; # Oracle allow multiple constraints declaration inside a single ALTER TABLE while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+CHECK\s*(\(.*?\))\s+(ENABLE|DISABLE|VALIDATE|NOVALIDATE|DEFERRABLE|INITIALLY|DEFERRED|USING\s+INDEX|\s+)+([^,]*)//is) { my $constname = $1; my $code = $2; my $states = $3; my $tbspace_move = $4; if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } my $validate = ''; $validate = ' NOT VALID' if ( $states =~ /NOVALIDATE/is); push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E CHECK $code$validate"); if ( $tbspace_move =~ /USING\s+INDEX\s+TABLESPACE\s+([^\s]+)/is) { if ($self->{use_tablespace}) { $tbspace_move = "ALTER INDEX $constname SET TABLESPACE " . lc($1); push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); } } elsif ($tbspace_move =~ /USING\s+INDEX\s+([^\s]+)/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); } } while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+FOREIGN\s+KEY\s*(\(.*?\)\s+REFERENCES\s+[^\s]+\s*\(.*?\))\s*([^,\)]+|$)//is) { my $constname = $1; my $other_def = $3; if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E FOREIGN KEY $2"); if ($other_def =~ /(ON\s+DELETE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; } if ($other_def =~ /(ON\s+UPDATE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; } my $validate = ''; $validate = ' NOT VALID' if ( $other_def =~ /NOVALIDATE/is); $self->{tables}{$tb_name}{alter_table}[-1] .= $validate; } # We can just have one primary key constraint if ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+PRIMARY KEY//is) { my $constname = lc($1); $tb_def =~ s/^[^\(]+//; $tb_def =~ s/\);$//s; if ( $tb_def =~ s/USING\s+INDEX\s+TABLESPACE\s+([^\s]+).*//s) { $tb_def =~ s/\s+$//; if ($self->{use_tablespace}) { my $tbspace_move = "ALTER INDEX $constname SET TABLESPACE $1"; push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); } push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT $constname PRIMARY KEY " . lc($tb_def)); } elsif ($tb_def =~ s/USING\s+INDEX\s+([^\s]+).*//s) { push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD PRIMARY KEY " . lc($tb_def)); $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); } elsif ($tb_def) { push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT $constname PRIMARY KEY " . lc($tb_def)); } if (!exists $self->{tables}{$tb_name}{table_info}{type}) { $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; $self->{tables}{$tb_name}{table_info}{num_rows} = 0; $tid++; $self->{tables}{$tb_name}{internal_id} = $tid; } } } else { print STDERR "[DEBUG] unhandled line: $content\n"; } } # Extract comments $self->read_comment_from_file(); } sub read_comment_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); my $tid = 0; while ($content =~ s/COMMENT\s+ON\s+TABLE\s+([^\s]+)\s+IS\s+'([^;]+);//is) { my $tb_name = $1; my $tb_comment = $2; $tb_name =~ s/"//g; $tb_comment =~ s/'\s*$//g; if (exists $self->{tables}{$tb_name}) { $self->{tables}{$tb_name}{table_info}{comment} = $tb_comment; } } while ($content =~ s/COMMENT\s+ON\s+COLUMN\s+(.*?)\s+IS\s*'([^;]+);//is) { my $tb_name = $1; my $tb_comment = $2; # register column name between double quote my $i = 0; my %col_name = (); # Get column name while ($tb_name =~ s/("[^"]+")/\%\%COLNAME$i\%\%/s) { $col_name{$i} = $1; $i++; } $tb_comment =~ s/'\s*$//g; if ($tb_name =~ s/\.([^\.]+)$//) { my $cname = $1; $tb_name =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; $tb_name =~ s/"//g; $cname =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; $cname =~ s/"//g; $cname =~ s/\./_/g; if (exists $self->{tables}{$tb_name}) { $self->{tables}{$tb_name}{column_comments}{"\L$cname\E"} = $tb_comment; } elsif (exists $self->{views}{$tb_name}) { $self->{views}{$tb_name}{column_comments}{"\L$cname\E"} = $tb_comment; } } else { $tb_name =~ s/\%\%COLNAME(\d+)\%\%/$col_name{$1}/sg; } } } sub read_view_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Clear content from comment and text constant for better parsing $self->_remove_comments(\$content); my $tid = 0; $content =~ s/\s+NO\s+FORCE\s+/ /igs; $content =~ s/\s+FORCE\s+EDITIONABLE\s+/ /igs; $content =~ s/\s+FORCE\s+/ /igs; $content =~ s/\s+EDITIONABLE\s+/ /igs; $content =~ s/\s+OR\s+REPLACE\s+/ /igs; $content =~ s/CREATE\s+VIEW\s+([^\s]+)\s+OF\s+(.*?)\s+AS\s+/CREATE VIEW $1 AS /isg; # Views with aliases while ($content =~ s/CREATE\s+VIEW\s+([^\s]+)\s*\((.*?)\)\s+AS\s+([^;]+)(;|$)//is) { my $v_name = $1; my $v_alias = $2; my $v_def = $3; $v_name =~ s/"//g; $tid++; $self->{views}{$v_name}{text} = $v_def; $self->{views}{$v_name}{iter} = $tid; # Remove constraint while ($v_alias =~ s/(,[^,\(]+\(.*)$//) {}; my @aliases = split(/\s*,\s*/, $v_alias); foreach (@aliases) { s/^\s+//; s/\s+$//; my @tmp = split(/\s+/); push(@{$self->{views}{$v_name}{alias}}, \@tmp); } } # Standard views while ($content =~ s/CREATE\s+VIEW\s+([^\s]+)\s+AS\s+([^;]+);//i) { my $v_name = $1; my $v_def = $2; $v_name =~ s/"//g; $tid++; $self->{views}{$v_name}{text} = $v_def; } # Extract comments $self->read_comment_from_file(); } sub read_grant_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Clear content from comment and text constant for better parsing $self->_remove_comments(\$content); my $tid = 0; # Extract grant information while ($content =~ s/GRANT\s+(.*?)\s+ON\s+([^\s]+)\s+TO\s+([^;]+)(\s+WITH GRANT OPTION)?;//i) { my $g_priv = $1; my $g_name = $2; $g_name =~ s/"//g; my $g_user = $3; my $g_option = $4; $g_priv =~ s/\s+//g; $tid++; $self->{grants}{$g_name}{type} = ''; push(@{$self->{grants}{$g_name}{privilege}{$g_user}}, split(/,/, $g_priv)); if ($g_priv =~ /EXECUTE/) { $self->{grants}{$g_name}{type} = 'PACKAGE BODY'; } else { $self->{grants}{$g_name}{type} = 'TABLE'; } } } sub read_trigger_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Clear content from comment and text constant for better parsing $self->_remove_comments(\$content); my $tid = 0; my $doloop = 1; my @triggers_decl = split(/(?:CREATE)?(?:\s+OR\s+REPLACE)?\s*(?:DEFINER=[^\s]+)?\s*\bTRIGGER(\s+|$)/is, $content); foreach $content (@triggers_decl) { my $t_name = ''; my $t_pos = ''; my $t_event = ''; my $tb_name = ''; my $trigger = ''; my $t_type = ''; my $t_referencing = ''; if ($content =~ s/^([^\s]+)\s+(BEFORE|AFTER|INSTEAD\s+OF)\s+(.*?)\s+ON\s+([^\s]+)\s+(.*)(\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,)[a-z0-9_\$"]*(?:;|$))//is) { $t_name = $1; $t_pos = $2; $t_event = $3; $tb_name = $4; $trigger = $5 . $6; $t_name =~ s/"//g; $tb_name =~ s/"//g; } elsif ($content =~ s/^([^\s]+)\s+(BEFORE|AFTER|INSTEAD|\s+|OF)((?:INSERT|UPDATE|DELETE|OR|\s+|OF)+\s+(?:.*?))*\s+ON\s+([^\s]+)\s+(.*)(\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,)[a-z0-9_\$"]*(?:;|$))//is) { $t_name = $1; $t_pos = $2; $t_event = $3; $tb_name = $4; $trigger = $5 . $6; $t_name =~ s/"//g; $tb_name =~ s/"//g; } next if (!$t_name || ! $tb_name); # Remove referencing clause, not supported by PostgreSQL < 10 if ($self->{pg_version} < 10) { $trigger =~ s/REFERENCING\s+(.*?)(FOR\s+EACH\s+)/$2/is; } elsif ($trigger =~ s/REFERENCING\s+(.*?)(FOR\s+EACH\s+)/$2/is) { $t_referencing = " REFERENCING $1"; } $t_referencing =~ s/REFERENCING\s+(NEW|OLD)\s+AS\s+(NEW|OLD)(\s+(NEW|OLD)\s+AS\s+(NEW|OLD))?//gsi; if ($trigger =~ s/^\s*(FOR\s+EACH\s+)(ROW|STATEMENT)\s*//is) { $t_type = $1 . $2; } my $t_when_cond = ''; if ($trigger =~ s/^\s*WHEN\s+(.*?)\s+((?:BEGIN|DECLARE|CALL).*)//is) { $t_when_cond = $1; $trigger = $2; if ($trigger =~ /^(BEGIN|DECLARE)/i) { ($trigger, $content) = &_get_plsql_code($trigger); } else { $trigger =~ s/([^;]+;)\s*(.*)/$1/; $content = $2; } } else { if ($trigger =~ /^(BEGIN|DECLARE)/i) { ($trigger, $content) = &_get_plsql_code($trigger); } } $tid++; # TRIGGER_NAME, TRIGGER_TYPE, TRIGGERING_EVENT, TABLE_NAME, TRIGGER_BODY, WHEN_CLAUSE, DESCRIPTION, ACTION_TYPE, OWNER $trigger =~ s/\bEND\s+[^\s]+\s+$/END/is; my $t_schema = ''; if ($t_name =~ s/^([^\.]+)\.//i) { if ($self->{export_schema} && ($self->{pg_schema} || $self->{schema} || $1)) { $t_schema = $self->{pg_schema} || $self->{schema} || $1; } } if ($tb_name =~ s/^([^\.]+)\.//i) { if ($self->{export_schema} && ($self->{pg_schema} || $self->{schema} || $1)) { $tb_name = ($self->{pg_schema} || $self->{schema} || $1) . '.' . $tb_name; } } my $when_event = ''; if ($t_when_cond) { $when_event = "$t_name\n$t_pos$t_referencing $t_event ON $tb_name\n$t_type"; } elsif ($t_referencing) { $when_event = $t_referencing; } $when_event =~ s/REFERENCING\s+(NEW|OLD)\s+AS\s+(NEW|OLD)\s+(NEW|OLD)\s+AS\s+(NEW|OLD)//gsi; push(@{$self->{triggers}}, [($t_name, $t_pos, $t_event, $tb_name, $trigger, $t_when_cond, $when_event, $t_type, $t_schema)]); } } sub read_sequence_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Clear content from comment and text constant for better parsing $self->_remove_comments(\$content, 1); $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; my $tid = 0; # Sequences while ($content =~ s/CREATE\s+SEQUENCE[\s]+([^\s;]+)\s*([^;]+);//i) { my $s_name = $1; my $s_def = $2; $s_name =~ s/"//g; $s_def =~ s/\s+/ /g; $tid++; my @seq_info = (); # Field of @seq_info # SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, LAST_NUMBER, CACHE_SIZE, CYCLE_FLAG, SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; push(@seq_info, $s_name); if ($s_def =~ /MINVALUE\s+([\-\d]+)/i) { push(@seq_info, $1); } else { push(@seq_info, ''); } if ($s_def =~ /MAXVALUE\s+([\-\d]+)/i) { if ($1 > 9223372036854775807) { push(@seq_info, 9223372036854775807); } else { push(@seq_info, $1); } } else { push(@seq_info, ''); } if ($s_def =~ /INCREMENT\s*(?:BY)?\s+([\-\d]+)/i) { push(@seq_info, $1); } else { push(@seq_info, 1); } if ($s_def =~ /START\s+WITH\s+([\-\d]+)/i) { push(@seq_info, $1); } else { push(@seq_info, ''); } if ($s_def =~ /CACHE\s+(\d+)/i) { push(@seq_info, $1); } else { push(@seq_info, ''); } if ($s_def =~ /NOCYCLE/i) { push(@seq_info, 'NO'); } else { push(@seq_info, 'YES'); } if ($s_name =~ /^([^\.]+)\./i) { push(@seq_info, $1); } else { push(@seq_info, ''); } push(@{$self->{sequences}{$s_name}}, @seq_info); } } sub read_tablespace_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); my @tbsps = split(/\s*;\s*/, $content); # tablespace without undo ones foreach $content (@tbsps) { $content .= ';'; if ($content =~ /CREATE\s+(?:BIGFILE|SMALLFILE)?\s*(?:TEMPORARY)?\s*TABLESPACE\s+([^\s;]+)\s*([^;]*);/is) { my $t_name = $1; my $t_def = $2; $t_name =~ s/"//g; if ($t_def =~ /(?:DATA|TEMP)FILE\s+'([^']+)'/is) { my $t_path = $1; $t_path =~ s/:/\//g; $t_path =~ s/\\/\//g; if (dirname($t_path) eq '.') { $t_path = 'change_tablespace_dir'; } else { $t_path = dirname($t_path); } # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME @{$self->{tablespaces}{TABLE}{$t_name}{$t_path}} = (); } } } } sub read_directory_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Directory while ($content =~ s/CREATE(?: OR REPLACE)?\s+DIRECTORY\s+([^\s]+)\s+AS\s+'([^']+)'\s*;//is) { my $d_name = uc($1); my $d_def = $2; $d_name =~ s/"//g; if ($d_def !~ /\/$/) { $d_def .= '/'; } $self->{directory}{$d_name}{path} = $d_def; } # Directory while ($content =~ s/GRANT\s+(.*?)ON\s+DIRECTORY\s+([^\s]+)\s+TO\s+([^;\s]+)\s*;//is) { my $d_grant = $1; my $d_name = uc($2); my $d_user = uc($3); $d_name =~ s/"//g; $d_user =~ s/"//g; $self->{directory}{$d_name}{grantee}{$d_user} = $d_grant; } } sub read_synonym_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Directory while ($content =~ s/CREATE(?: OR REPLACE)?(?: PUBLIC)?\s+SYNONYM\s+([^\s]+)\s+FOR\s+([^;\s]+)\s*;//is) { my $s_name = uc($1); my $s_def = $2; $s_name =~ s/"//g; $s_def =~ s/"//g; if ($s_name =~ s/^([^\.]+)\.//) { $self->{synonyms}{$s_name}{owner} = $1; } else { $self->{synonyms}{$s_name}{owner} = $self->{schema}; } if ($s_def =~ s/@(.*)//) { $self->{synonyms}{$s_name}{dblink} = $1; } if ($s_def =~ s/^([^\.]+)\.//) { $self->{synonyms}{$s_name}{table_owner} = $1; } $self->{synonyms}{$s_name}{table_name} = $s_def; } } sub read_dblink_from_file { my $self = shift; # Load file in a single string my $content = $self->_get_ddl_from_file(); # Directory while ($content =~ s/CREATE(?: SHARED)?(?: PUBLIC)?\s+DATABASE\s+LINK\s+([^\s]+)\s+CONNECT TO\s+([^\s]+)\s*([^;]+);//is) { my $d_name = $1; my $d_user = $2; my $d_auth = $3; $d_name =~ s/"//g; $d_user =~ s/"//g; $self->{dblink}{$d_name}{owner} = $self->{shema}; $self->{dblink}{$d_name}{user} = $d_user; $self->{dblink}{$d_name}{username} = $self->{pg_user} || $d_user; if ($d_auth =~ s/USING\s+([^\s]+)//) { $self->{dblink}{$d_name}{host} = $1; $self->{dblink}{$d_name}{host} =~ s/'//g; } if ($d_auth =~ s/IDENTIFIED\s+BY\s+([^\s]+)//) { $self->{dblink}{$d_name}{password} = $1; } if ($d_auth =~ s/AUTHENTICATED\s+BY\s+([^\s]+)\s+IDENTIFIED\s+BY\s+([^\s]+)//) { $self->{dblink}{$d_name}{user} = $1; $self->{dblink}{$d_name}{password} = $2; $self->{dblink}{$d_name}{username} = $self->{pg_user} || $1; } } # Directory while ($content =~ s/CREATE(?: SHARED)?(?: PUBLIC)?\s+DATABASE\s+LINK\s+([^\s]+)\s+USING\s+([^;]+);//is) { my $d_name = $1; my $d_conn = $2; $d_name =~ s/"//g; $d_conn =~ s/'//g; $self->{dblink}{$d_name}{owner} = $self->{shema}; $self->{dblink}{$d_name}{host} = $d_conn; } } =head2 _views This function is used to retrieve all views information. Sets the main hash of the views definition $self->{views}. Keys are the names of all views retrieved from the current database and values are the text definitions of the views. It then sets the main hash as follows: # Definition of the view $self->{views}{$table}{text} = $lview_infos{$table}; =cut sub _views { my ($self) = @_; # Get all views information $self->logit("Retrieving views information...\n", 1); my %view_infos = $self->_get_views(); # Retrieve comment of each columns my %columns_comments = $self->_column_comments(); foreach my $view (keys %columns_comments) { next if (!exists $view_infos{$view}); foreach my $c (keys %{$columns_comments{$view}}) { $self->{views}{$view}{column_comments}{$c} = $columns_comments{$view}{$c}; } } my $i = 1; foreach my $view (sort keys %view_infos) { $self->logit("[$i] Scanning $view...\n", 1); $self->{views}{$view}{text} = $view_infos{$view}{text}; $self->{views}{$view}{owner} = $view_infos{$view}{owner}; $self->{views}{$view}{check_option} = $view_infos{$view}{check_option}; $self->{views}{$view}{updatable} = $view_infos{$view}{updatable}; $self->{views}{$view}{iter} = $view_infos{$view}{iter} if (exists $view_infos{$view}{iter}); $self->{views}{$view}{comment} = $view_infos{$view}{comment}; # Retrieve also aliases from views $self->{views}{$view}{alias} = $view_infos{$view}{alias}; $i++; } } =head2 _materialized_views This function is used to retrieve all materialized views information. Sets the main hash of the views definition $self->{materialized_views}. Keys are the names of all materialized views retrieved from the current database and values are the text definitions of the views. It then sets the main hash as follows: # Definition of the materialized view $self->{materialized_views}{text} = $mview_infos{$view}; =cut sub _materialized_views { my ($self) = @_; # Get all views information $self->logit("Retrieving materialized views information...\n", 1); my %mview_infos = $self->_get_materialized_views(); my $i = 1; foreach my $table (sort keys %mview_infos) { $self->logit("[$i] Scanning $table...\n", 1); $self->{materialized_views}{$table}{text} = $mview_infos{$table}{text}; $self->{materialized_views}{$table}{updatable}= $mview_infos{$table}{updatable}; $self->{materialized_views}{$table}{refresh_mode}= $mview_infos{$table}{refresh_mode}; $self->{materialized_views}{$table}{refresh_method}= $mview_infos{$table}{refresh_method}; $self->{materialized_views}{$table}{no_index}= $mview_infos{$table}{no_index}; $self->{materialized_views}{$table}{rewritable}= $mview_infos{$table}{rewritable}; $self->{materialized_views}{$table}{build_mode}= $mview_infos{$table}{build_mode}; $self->{materialized_views}{$table}{owner}= $mview_infos{$table}{owner}; $i++; } # Retrieve index informations if (scalar keys %mview_infos) { my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}); foreach my $tb (keys %{$indexes}) { next if (!exists $self->{materialized_views}{$tb}); %{$self->{materialized_views}{$tb}{indexes}} = %{$indexes->{$tb}}; } foreach my $tb (keys %{$idx_type}) { next if (!exists $self->{materialized_views}{$tb}); %{$self->{materialized_views}{$tb}{idx_type}} = %{$idx_type->{$tb}}; } } } =head2 _tablespaces This function is used to retrieve all Oracle Tablespaces information. Sets the main hash $self->{tablespaces}. =cut sub _tablespaces { my ($self) = @_; $self->logit("Retrieving tablespaces information...\n", 1); $self->{tablespaces} = $self->_get_tablespaces(); $self->{list_tablespaces} = $self->_list_tablespaces(); } =head2 _partitions This function is used to retrieve all Oracle partition information. Sets the main hash $self->{partition}. =cut sub _partitions { my ($self) = @_; $self->logit("Retrieving partitions information...\n", 1); ($self->{partitions}, $self->{partitions_default}) = $self->_get_partitions(); ($self->{subpartitions}, $self->{subpartitions_default}) = $self->_get_subpartitions(); # Get partition list meta information %{ $self->{partitions_list} } = $self->_get_partitioned_table(); %{ $self->{subpartitions_list} } = $self->_get_subpartitioned_table(); # Look for main table indexes to reproduce them on partition my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}, 0); foreach my $tb (keys %{$indexes}) { %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; } foreach my $tb (keys %{$idx_type}) { %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; } foreach my $tb (keys %{$idx_tbsp}) { %{$self->{tables}{$tb}{idx_tbsp}} = %{$idx_tbsp->{$tb}}; } foreach my $tb (keys %{$uniqueness}) { %{$self->{tables}{$tb}{uniqueness}} = %{$uniqueness->{$tb}}; } # Retrieve all unique keys informations my %unique_keys = $self->_unique_key('',$self->{schema}); foreach my $tb (keys %unique_keys) { foreach my $c (keys %{$unique_keys{$tb}}) { $self->{tables}{$tb}{unique_key}{$c} = $unique_keys{$tb}{$c}; } } } =head2 _dblinks This function is used to retrieve all Oracle dblinks information. Sets the main hash $self->{dblink}. =cut sub _dblinks { my ($self) = @_; $self->logit("Retrieving dblinks information...\n", 1); %{$self->{dblink}} = $self->_get_dblink(); } =head2 _directories This function is used to retrieve all Oracle directories information. Sets the main hash $self->{directory}. =cut sub _directories { my ($self) = @_; $self->logit("Retrieving directories information...\n", 1); %{$self->{directory}} = $self->_get_directory(); } sub get_replaced_tbname { my ($self, $tmptb) = @_; if (exists $self->{replaced_tables}{"\L$tmptb\E"} && $self->{replaced_tables}{"\L$tmptb\E"}) { $self->logit("\tReplacing table $tmptb as " . $self->{replaced_tables}{lc($tmptb)} . "...\n", 1); $tmptb = $self->{replaced_tables}{lc($tmptb)}; } $tmptb = $self->quote_object_name($tmptb); return $tmptb; } sub get_tbname_with_suffix { my ($self, $tmptb, $suffix) = @_; return $self->quote_object_name($tmptb . $suffix); } sub _export_table_data { my ($self, $table, $part_name, $subpart, $pos, $p, $dirprefix, $sql_header) = @_; if ($subpart) { $self->logit("(pid: $$) Exporting data of subpartition $subpart of partition $part_name from table $table...\n", 1); } elsif ($part_name) { $self->logit("(pid: $$) Exporting data of partition $part_name from table $table...\n", 1); } else { $self->logit("(pid: $$) Exporting data from table $table...\n", 1); } # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # register the column list and data type in dedicated structs @{$self->{data_cols}{$table}} = (); @{$self->{tables}{$table}{field_name}} = (); @{$self->{tables}{$table}{field_type}} = (); foreach my $k (sort {$self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]} keys %{$self->{tables}{$table}{column_info}}) { push(@{$self->{data_cols}{$table}}, $k); push(@{$self->{tables}{$table}{field_name}}, $self->{tables}{$table}{column_info}{$k}[0]); push(@{$self->{tables}{$table}{field_type}}, $self->{tables}{$table}{column_info}{$k}[1]); } # Open output file $self->data_dump($sql_header, $table) if (!$self->{pg_dsn} && $self->{file_per_table}); my $total_record = 0; # When copy freeze is required, force a transaction with a truncate if ($self->{copy_freeze} && !$self->{pg_dsn}) { $self->{truncate_table} = 1; if ($self->{file_per_table}) { $self->data_dump("BEGIN;\n", $table); } else { $self->dump("\nBEGIN;\n"); } } else { $self->{copy_freeze} = ''; } # Open a new connection to PostgreSQL destination with parallel table export my $local_dbh = undef; if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { $local_dbh = $self->_send_to_pgdb(); } else { $local_dbh = $self->{dbhdest}; } if ($self->{global_delete} || exists $self->{delete}{"\L$table\E"}) { my $delete_clause = ''; my $delete_clause_start = "DELETE"; if ($self->{datadiff}) { $delete_clause_start = "INSERT INTO " . $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}) . " SELECT *"; } if (exists $self->{delete}{"\L$table\E"} && $self->{delete}{"\L$table\E"}) { $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{delete}{"\L$table\E"} . ";"; $self->logit("\tApplying DELETE clause on table: " . $self->{delete}{"\L$table\E"} . "\n", 1); } elsif ($self->{global_delete}) { $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{global_delete} . ";"; $self->logit("\tApplying DELETE global clause: " . $self->{global_delete} . "\n", 1); } if ($delete_clause) { if ($self->{pg_dsn}) { $self->logit("Deleting from table $table...\n", 1); my $s = $local_dbh->do("$delete_clause") or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } else { if ($self->{file_per_table}) { $self->data_dump("$delete_clause\n", $table); } else { $self->dump("\n$delete_clause\n"); } } } } # Set parent table name to compose partition name my $ptmptb = $table; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $ptmptb = $self->{replaced_tables}{lc($table)}; } my $tbpart_name = ''; if ($part_name) { $tbpart_name = $part_name; $tbpart_name = $ptmptb . '_part' . $pos if ($self->{rename_partition}); if ($self->{rename_partition} && $part_name eq 'default') { $tbpart_name = $table . '_part_default'; } } my $sub_tb_name = ''; if ($subpart) { $sub_tb_name = $subpart; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any $sub_tb_name = $ptmptb . '_part' . $pos . '_subpart' . $p if ($self->{rename_partition}); if ($self->{rename_partition} && $subpart eq 'default') { $sub_tb_name = $tbpart_name . '_subpart_default'; } } # Set search path my $search_path = $self->set_search_path(); # Add table truncate order if there's no global DELETE clause or one specific to the current table if ($self->{truncate_table} && !$self->{global_delete} && !exists $self->{delete}{"\L$table\E"}) { my $truncate_order = "TRUNCATE TABLE "; if ($subpart) { $truncate_order .= $sub_tb_name; } elsif ($part_name) { $truncate_order .= $tbpart_name; } else { $truncate_order .= $tmptb; } if ($self->{pg_dsn} && !$self->{oracle_speed}) { if ($search_path) { $self->logit("Setting search_path using: $search_path...\n", 1); $local_dbh->do($search_path) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } $self->logit("$truncate_order...\n", 1); my $s = $local_dbh->do($truncate_order) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } else { my $head = "SET client_encoding TO '\U$self->{client_encoding}\E';\n"; $head .= "SET synchronous_commit TO off;\n" if (!$self->{synchronous_commit}); if ($self->{file_per_table}) { $self->data_dump("$head$search_path\n$truncate_order;\n", $table); } else { $self->dump("\n$head$search_path\n$truncate_order;\n"); } } } else { my $head = "SET client_encoding TO '\U$self->{client_encoding}\E';\n"; $head .= "SET synchronous_commit TO off;\n" if (!$self->{synchronous_commit}); if ($self->{file_per_table}) { $self->data_dump("$head$search_path\n", $table); } else { $self->dump("\n$head$search_path\n"); } } # With partitioned table, load data direct from table partition or subpartition if ($subpart) { if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists if ($self->file_exists("$dirprefix${sub_tb_name}_$self->{output}")) { # close the connection with parallel table export if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { $local_dbh->disconnect() if (defined $local_dbh); } return $total_record; } } $self->logit("Dumping sub partition table $table -> $part_name -> $subpart...\n", 1); $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $subpart, 1, $tbpart_name, $sub_tb_name); # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $subpart, $table); } elsif ($part_name) { if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists if ($self->file_exists("$dirprefix${tbpart_name}_$self->{output}")) { # close the connection with parallel table export if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { $local_dbh->disconnect() if (defined $local_dbh); } return $total_record; } } $self->logit("Dumping partition table $table -> $part_name...\n", 1); $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $part_name, 0, $tbpart_name); # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $part_name, $table); } elsif (exists $self->{partitions}{$table}) { foreach my $pos (sort {$self->{partitions}{$table}{$a} <=> $self->{partitions}{$table}{$b}} keys %{$self->{partitions}{$table}}) { my $part_name = $self->{partitions}{$table}{$pos}{name}; my $tbpart_name = $part_name; $tmptb = $table; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $tmptb = $self->{replaced_tables}{lc($table)}; } $tbpart_name = $tmptb . '_part' . $pos if ($self->{rename_partition}); next if ($self->{allow_partition} && !grep($_ =~ /^$tbpart_name$/i, @{$self->{allow_partition}})); if (exists $self->{subpartitions}{$table}{$part_name}) { foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) { my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); my $sub_tb_name = $subpart; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any $sub_tb_name = $tmptb . '_part' . $pos . '_subpart' . $p if ($self->{rename_partition}); if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists next if ($self->file_exists("$dirprefix${sub_tb_name}_$self->{output}")); } if ($#{$self->{tables}{$table}{field_name}} < 0) { $self->logit("Table $table has no column defined, skipping...\n", 1); next; } $self->logit("Dumping sub partition table $table ($subpart)...\n", 1); $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $subpart, 1, $tbpart_name, $sub_tb_name); # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $subpart, $table); } # Now load content of the default subpartition table if ($self->{subpartitions_default}{$table}{$part_name}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}{name}$/i, @{$self->{allow_partition}})) { if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists if (!$self->file_exists("$dirprefix$self->{subpartitions_default}{$table}{$part_name}{name}_$self->{output}")) { if ($self->{rename_partition}) { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}{name}, 1, $tbpart_name, $tbpart_name . '_subpart_default'); } else { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}{name}, 1, $part_name, $self->{subpartitions_default}{$table}{$part_name}{name}); } } } else { if ($self->{rename_partition}) { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}{name}, 1, $tbpart_name, $tbpart_name . '_subpart_default'); } else { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}{name}, 1, $part_name, $self->{subpartitions_default}{$table}{$part_name}{name}); } } } # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $self->{subpartitions_default}{$table}{$part_name}{name}, $table); } } else { if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists next if ($self->file_exists("$dirprefix${tbpart_name}_$self->{output}")); } $self->logit("Dumping partition table $table ($part_name)...\n", 1); $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $part_name, 0, $tbpart_name); # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $part_name, $table); } } # Now load content of the default partition table if (exists $self->{partitions_default}{$table}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}{name}$/i, @{$self->{allow_partition}})) { my $tbpart_name = $self->{partitions_default}{$table}{name}; my $tmptb = $table; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $tmptb = $self->{replaced_tables}{lc($table)}; } $tbpart_name = $tmptb . '_part_default' if ($self->{rename_partition}); if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists if (!$self->file_exists("$dirprefix$self->{partitions_default}{$table}{name}_$self->{output}")) { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{partitions_default}{$table}{name}, 0, $tbpart_name); } } else { $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{partitions_default}{$table}{name}, 0, $tbpart_name); } # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $self->{partitions_default}{$table}{name}, $table); } } } else { # Do not dump data if the table has no column if ($#{$self->{tables}{$table}{field_name}} < 0) { $self->logit("Table $table has no column defined, skipping...\n", 1); } else { $total_record = $self->_dump_table($dirprefix, $sql_header, $table); } } # close the connection with parallel table export if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { $local_dbh->disconnect() if (defined $local_dbh); } # Rename temporary filename into final name $self->rename_dump_partfile($dirprefix, $table) if (!$self->{oracle_speed}); return $total_record; } sub _export_fdw_table_data { my ($self, $table, $dirprefix, $sql_header) = @_; $self->logit("FATAL: foreign data export requires that PG_DSN to be set \n", 0, 1) if (!$self->{pg_dsn}); $self->logit("Exporting data of table $table using foreign table...\n", 1); # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); my $total_record = 0; $self->{copy_freeze} = ''; # Open a new connection to PostgreSQL destination with parallel table export my $local_dbh = undef; if ($self->{parallel_tables} > 1 && $self->{pg_dsn}) { $local_dbh = $self->_send_to_pgdb(); } else { $local_dbh = $self->{dbhdest}; } if ($self->{global_delete} || exists $self->{delete}{"\L$table\E"}) { my $delete_clause = ''; my $delete_clause_start = "DELETE"; if (exists $self->{delete}{"\L$table\E"} && $self->{delete}{"\L$table\E"}) { $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{delete}{"\L$table\E"} . ";"; $self->logit("\tApplying DELETE clause on table: " . $self->{delete}{"\L$table\E"} . "\n", 1); } elsif ($self->{global_delete}) { $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{global_delete} . ";"; $self->logit("\tApplying DELETE global clause: " . $self->{global_delete} . "\n", 1); } if ($delete_clause) { $self->logit("Deleting from table $table...\n", 1); my $s = $local_dbh->do("$delete_clause") or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } } # Add table truncate order if there's no global DELETE clause or one specific to the current table if ($self->{truncate_table} && !$self->{global_delete} && !exists $self->{delete}{"\L$table\E"}) { # Set search path my $search_path = $self->set_search_path(); if (!$self->{oracle_speed}) { if ($search_path) { $self->logit("Setting search_path using: $search_path...\n", 1); $local_dbh->do($search_path) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } $self->logit("Truncating table $tmptb...\n", 1); my $s = $local_dbh->do("TRUNCATE TABLE $tmptb;") or $self->logit("FATAL: " . $local_dbh->errstr . ". SQL: TRUNCATE TABLE $tmptb;\n", 0, 1); } } $total_record = $self->_dump_fdw_table($dirprefix, $sql_header, $table, $local_dbh); # close the connection with parallel table export if ($self->{parallel_tables} > 1) { $local_dbh->disconnect() if (defined $local_dbh); } return $total_record; } sub rename_dump_partfile { my ($self, $dirprefix, $partname, $tbl) = @_; my $filename = "${dirprefix}tmp_${tbl}_${partname}_$self->{output}"; my $filedest = "${dirprefix}${tbl}_${partname}_$self->{output}"; if (!$tbl) { $filename = "${dirprefix}tmp_${partname}_$self->{output}"; $filedest = "${dirprefix}${partname}_$self->{output}"; } if (-e $filename) { $self->logit("Renaming temporary file $filename into $filedest\n", 1); rename($filename, $filedest); } } sub set_refresh_count { my $count = shift; return 500 if ($count > 10000); return 100 if ($count > 1000); return 10 if ($count > 100); return 1; } sub translate_function { my ($self, $i, $num_total_function, %functions) = @_; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); # Clear memory in multiprocess mode if ($self->{jobs} > 1) { $self->{functions} = (); $self->{procedures} = (); } my $t0 = Benchmark->new; my $sql_output = ''; my $lsize = 0; my $lcost = 0; my $fct_count = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_function); foreach my $fct (sort keys %functions) { if (!$self->{quiet} && !$self->{debug} && ($fct_count % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i+1, $num_total_function, 25, '=', 'functions', "generating $fct" ), "\r"; } $fct_count++; $self->logit("Dumping function $fct...\n", 1); if ($self->{file_per_function}) { my $f = "$dirprefix${fct}_$self->{output}"; $f = "${fct}_$self->{output}" if ($self->{psql_relative_path}); $f =~ s/\.(?:gz|bz2)$//i; $self->dump("\\i$self->{psql_relative_path} $f\n"); $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($fct), "$dirprefix${fct}_$self->{output}"); } else { $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($fct), "$dirprefix$self->{output}"); } my $fhdl = undef; $self->_remove_comments(\$functions{$fct}{text}); $lsize = length($functions{$fct}{text}); if ($self->{file_per_function}) { $self->logit("Dumping to one file per function : ${fct}_$self->{output}\n", 1); $fhdl = $self->open_export_file("${fct}_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); } if ($self->{plsql_pgsql}) { my $sql_f = ''; if ($self->{is_mysql}) { $sql_f = $self->_convert_function($functions{$fct}{owner}, $functions{$fct}{text}, $fct); } else { $sql_f = $self->_convert_function($functions{$fct}{owner}, $functions{$fct}{text}); } if ( $sql_f ) { $sql_output .= $sql_f . "\n\n"; if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_f, 'FUNCTION'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION'}; $lcost += $cost; $self->logit("Function ${fct} estimated cost: $cost\n", 1); $sql_output .= "-- Function ${fct} estimated cost: $cost\n"; foreach (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$_}); $sql_output .= "\t-- $_ => $cost_detail{$_}"; if (!$self->{is_mysql}) { $sql_output .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_SCORE{$_}); } else { $sql_output .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_}); } $sql_output .= "\n"; } if ($self->{jobs} > 1) { my $tfh = $self->append_export_file($dirprefix . 'temp_cost_file.dat', 1); flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; $tfh->print("${fct}:$lsize:$lcost\n"); $self->close_export_file($tfh, 1); } } } } else { $sql_output .= $functions{$fct}{text} . "\n\n"; } $self->_restore_comments(\$sql_output); if ($self->{plsql_pgsql}) { $sql_output =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; } my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; $sql_header .= "-- Copyright 2000-2025 Gilles DAROLD. All rights reserved.\n"; $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; if ($self->{client_encoding}) { $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n\n"; } if ($self->{type} ne 'TABLE') { $sql_header .= $self->set_search_path(); } $sql_header .= "\\set ON_ERROR_STOP ON\n\n" if ($self->{stop_on_error}); $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); $sql_header = '' if ($self->{no_header}); if ($self->{file_per_function}) { $self->dump($sql_header . $sql_output, $fhdl); $self->close_export_file($fhdl); $sql_output = ''; } } my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Translating of $fct_count functions took: " . timestr($td) . "\n", 1); return ($sql_output, $lsize, $lcost); } sub _replace_declare_var { my ($self, $code) = @_; if ($$code =~ s/\b(DECLARE\s+(?:.*?)\s+BEGIN)/\%DECLARE\%/is) { my $declare = $1; # Collect user defined exception while ($declare =~ s/\b([^\s]+)\s+EXCEPTION\s*;//i) { my $e = lc($1); if (!exists $Ora2Pg::PLSQL::EXCEPTION_MAP{"\U$e\L"} && !grep(/^$e$/, values %Ora2Pg::PLSQL::EXCEPTION_MAP) && !exists $self->{custom_exception}{$e}) { $self->{custom_exception}{$e} = $self->{exception_id}++; } } $declare =~ s/PRAGMA\s+EXCEPTION_INIT[^;]*;//igs; if ($self->{is_mysql}) { ($$code, $declare) = Ora2Pg::MySQL::replace_mysql_variables($self, $$code, $declare); } $$code =~ s/\%DECLARE\%/$declare/is; } elsif ($self->{is_mysql}) { ($$code, $declare) = Ora2Pg::MySQL::replace_mysql_variables($self, $$code, $declare); $$code = "DECLARE\n" . $declare . "\n" . $$code if ($declare); } # Replace call to raise exception foreach my $e (keys %{$self->{custom_exception}}) { $$code =~ s/\bRAISE\s+$e\b/RAISE EXCEPTION '$e' USING ERRCODE = '$self->{custom_exception}{$e}'/igs; $$code =~ s/(\s+(?:WHEN|OR)\s+)$e\s+/$1SQLSTATE '$self->{custom_exception}{$e}' /igs; } } # Routine used to save the file to update in pass2 of translation sub save_filetoupdate_list { my ($self, $pname, $ftcname, $file_name) = @_; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $tfh = $self->append_export_file($dirprefix . 'temp_pass2_file.dat', 1); flock($tfh, 2) || die "FATAL: can't lock file temp_pass2_file.dat\n"; $tfh->print("${pname}:${ftcname}:$file_name\n"); $self->close_export_file($tfh, 1); } =head2 _set_file_header Returns a string containing the common header of each output file. =cut sub _set_file_header { my $self = shift(); return '' if ($self->{no_header}); my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; $sql_header .= "-- Copyright 2000-2025 Gilles DAROLD. All rights reserved.\n"; $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; if ($self->{client_encoding}) { $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n\n"; } if ($self->{type} ne 'TABLE') { $sql_header .= $self->set_search_path(); } $sql_header .= "\\set ON_ERROR_STOP ON\n\n" if ($self->{stop_on_error}); $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); return $sql_header; } =head2 export_view Export Oracle view into PostgreSQL compatible SQL statements. =cut sub export_view { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add views definition...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_view_from_file(); } my $nothing = 0; $self->dump($sql_header); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $i = 1; my $num_total_view = scalar keys %{$self->{views}}; %ordered_views = %{$self->{views}}; my $count_view = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_view); foreach my $view (sort sort_view_by_iter keys %ordered_views) { $self->logit("\tAdding view $view...\n", 1); if (!$self->{quiet} && !$self->{debug} && ($count_view % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_view, 25, '=', 'views', "generating $view" ), "\r"; } $count_view++; my $fhdl = undef; if ($self->{file_per_table}) { my $file_name = "$dirprefix${view}_$self->{output}"; $file_name = "${view}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $self->dump("\\i$self->{psql_relative_path} '$file_name'\n"); $self->logit("Dumping to one file per view : ${view}_$self->{output}\n", 1); $fhdl = $self->open_export_file("${view}_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), $file_name); } else { $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), "$dirprefix$self->{output}"); } $self->_remove_comments(\$self->{views}{$view}{text}); if (!$self->{pg_supports_checkoption}) { $self->{views}{$view}{text} =~ s/\s*WITH\s+CHECK\s+OPTION//is; } # Remove unsupported definitions from the ddl statement $self->{views}{$view}{text} =~ s/\s*WITH\s+READ\s+ONLY//is; $self->{views}{$view}{text} =~ s/\s*OF\s+([^\s]+)\s+(WITH|UNDER)\s+[^\)]+\)//is; $self->{views}{$view}{text} =~ s/\s*OF\s+XMLTYPE\s+[^\)]+\)//is; $self->{views}{$view}{text} = $self->_format_view($view, $self->{views}{$view}{text}); my $tmpv = $view; if (exists $self->{replaced_tables}{"\L$tmpv\E"} && $self->{replaced_tables}{"\L$tmpv\E"}) { $self->logit("\tReplacing table $tmpv as " . $self->{replaced_tables}{lc($tmpv)} . "...\n", 1); $tmpv = $self->{replaced_tables}{lc($tmpv)}; } if ($self->{export_schema} && !$self->{schema} && ($tmpv =~ /^([^\.]+)\./) ) { $sql_output .= $self->set_search_path($1) . "\n"; } $tmpv = $self->quote_object_name($tmpv); if (!@{$self->{views}{$view}{alias}}) { $sql_output .= "CREATE$self->{create_or_replace} VIEW $tmpv AS "; $sql_output .= $self->{views}{$view}{text}; $sql_output .= ';' if ($sql_output !~ /;\s*$/s); $sql_output .= "\n"; if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $self->{views}{$view}{text}, 'VIEW'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; $cost_value += $cost; $sql_output .= "\n-- Estimed cost of view [ $view ]: " . sprintf("%2.2f", $cost); } $sql_output .= "\n"; } else { $sql_output .= "CREATE$self->{create_or_replace} VIEW $tmpv ("; my $count = 0; my %col_to_replace = (); foreach my $d (@{$self->{views}{$view}{alias}}) { if ($count == 0) { $count = 1; } else { $sql_output .= ", "; } # Change column names my $fname = $d->[0]; if (exists $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$d->[0]\E as " . $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"}; } $sql_output .= $self->quote_object_name($fname); } $sql_output .= ") AS " . $self->{views}{$view}{text}; $sql_output .= ';' if ($sql_output !~ /;\s*$/s); $sql_output .= "\n"; if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $self->{views}{$view}{text}, 'VIEW'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; $cost_value += $cost; $sql_output .= "\n-- Estimed cost of view [ $view ]: " . sprintf("%2.2f", $cost); } $sql_output .= "\n"; } if ($self->{force_owner}) { my $owner = $self->{views}{$view}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER VIEW $tmpv OWNER TO " . $self->quote_object_name($owner) . ";\n"; } # Add comments on view and columns if (!$self->{disable_comment}) { if ($self->{views}{$view}{comment}) { $sql_output .= "COMMENT ON VIEW $tmpv "; $self->{views}{$view}{comment} =~ s/'/''/gs; $sql_output .= " IS E'" . $self->{views}{$view}{comment} . "';\n\n"; } foreach my $f (sort { lc{$a} cmp lc($b) } keys %{$self->{views}{$view}{column_comments}}) { next unless $self->{views}{$view}{column_comments}{$f}; $self->{views}{$view}{column_comments}{$f} =~ s/'/''/gs; # Change column names my $fname = $f; if (exists $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"} && $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"}) { $fname = $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"}; } $sql_output .= "COMMENT ON COLUMN " . $self->quote_object_name($tmpv) . '.' . $self->quote_object_name($fname) . " IS E'" . $self->{views}{$view}{column_comments}{$f} . "';\n"; } } if ($self->{file_per_table}) { $self->dump($sql_header . $sql_output, $fhdl); $self->_restore_comments(\$sql_output); $self->close_export_file($fhdl); $sql_output = ''; } $nothing++; $i++; } %ordered_views = (); if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_view, 25, '=', 'views', 'end of output.'), "\n"; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } else { $sql_output .= "\n"; } $self->dump($sql_output); return; } =head2 export_mview Export Oracle materialized view into PostgreSQL compatible SQL statements. =cut sub export_mview { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add materialized views definition...\n", 1); my $nothing = 0; $self->dump($sql_header) if ($self->{file_per_table} && !$self->{pg_dsn}); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); if ($self->{plsql_pgsql} && !$self->{pg_supports_mview}) { $sql_header .= "DROP TABLE $self->{pg_supports_ifexists} materialized_views;\n" if ($self->{drop_if_exists}); my $sqlout = qq{ $sql_header CREATE TABLE materialized_views ( mview_name text NOT NULL PRIMARY KEY, view_name text NOT NULL, iname text, last_refresh TIMESTAMP WITH TIME ZONE ); CREATE OR REPLACE FUNCTION create_materialized_view(text, text, text) RETURNS VOID AS \$\$ DECLARE mview ALIAS FOR \$1; -- name of the materialized view to create vname ALIAS FOR \$2; -- name of the related view iname ALIAS FOR \$3; -- name of the colum of mview to used as unique key entry materialized_views%ROWTYPE; BEGIN EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ' || quote_literal(mview) || '' INTO entry; IF entry.iname IS NOT NULL THEN RAISE EXCEPTION 'Materialized view % already exist.', mview; END IF; EXECUTE 'REVOKE ALL ON ' || quote_ident(vname) || ' FROM PUBLIC'; EXECUTE 'GRANT SELECT ON ' || quote_ident(vname) || ' TO PUBLIC'; EXECUTE 'CREATE TABLE ' || quote_ident(mview) || ' AS SELECT * FROM ' || quote_ident(vname); EXECUTE 'REVOKE ALL ON ' || quote_ident(mview) || ' FROM PUBLIC'; EXECUTE 'GRANT SELECT ON ' || quote_ident(mview) || ' TO PUBLIC'; INSERT INTO materialized_views (mview_name, view_name, iname, last_refresh) VALUES ( quote_literal(mview), quote_literal(vname), quote_literal(iname), CURRENT_TIMESTAMP ); IF iname IS NOT NULL THEN EXECUTE 'CREATE INDEX ' || quote_ident(mview) || '_' || quote_ident(iname) || '_idx ON ' || quote_ident(mview) || '(' || quote_ident(iname) || ')'; END IF; RETURN; END \$\$ SECURITY DEFINER LANGUAGE plpgsql; CREATE OR REPLACE FUNCTION drop_materialized_view(text) RETURNS VOID AS \$\$ DECLARE mview ALIAS FOR \$1; entry materialized_views%ROWTYPE; BEGIN EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ''' || quote_literal(mview) || '''' INTO entry; IF entry.iname IS NULL THEN RAISE EXCEPTION 'Materialized view % does not exist.', mview; END IF; IF entry.iname IS NOT NULL THEN EXECUTE 'DROP INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx'; END IF; EXECUTE 'DROP TABLE ' || quote_ident(mview); EXECUTE 'DELETE FROM materialized_views WHERE mview_name=''' || quote_literal(mview) || ''''; RETURN; END \$\$ SECURITY DEFINER LANGUAGE plpgsql ; CREATE OR REPLACE FUNCTION refresh_full_materialized_view(text) RETURNS VOID AS \$\$ DECLARE mview ALIAS FOR \$1; entry materialized_views%ROWTYPE; BEGIN EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ''' || quote_literal(mview) || '''' INTO entry; IF entry.iname IS NULL THEN RAISE EXCEPTION 'Materialized view % does not exist.', mview; END IF; IF entry.iname IS NOT NULL THEN EXECUTE 'DROP INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx'; END IF; EXECUTE 'TRUNCATE ' || quote_ident(mview); EXECUTE 'INSERT INTO ' || quote_ident(mview) || ' SELECT * FROM ' || entry.view_name; EXECUTE 'UPDATE materialized_views SET last_refresh=CURRENT_TIMESTAMP WHERE mview_name=''' || quote_literal(mview) || ''''; IF entry.iname IS NOT NULL THEN EXECUTE 'CREATE INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx ON ' || quote_ident(mview) || '(' || entry.iname || ')'; END IF; RETURN; END \$\$ SECURITY DEFINER LANGUAGE plpgsql ; }; $self->dump($sqlout); } my $i = 1; my $num_total_mview = scalar keys %{$self->{materialized_views}}; my $count_mview = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_mview); foreach my $view (sort { $a cmp $b } keys %{$self->{materialized_views}}) { $self->logit("\tAdding materialized view $view...\n", 1); if (!$self->{quiet} && !$self->{debug} && ($count_mview % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_mview, 25, '=', 'materialized views', "generating $view" ), "\r"; } $count_mview++; my $fhdl = undef; if ($self->{file_per_table} && !$self->{pg_dsn}) { my $file_name = "$dirprefix${view}_$self->{output}"; $file_name = "${view}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $self->dump("\\i$self->{psql_relative_path} '$file_name'\n"); $self->logit("Dumping to one file per materialized view : ${view}_$self->{output}\n", 1); $fhdl = $self->open_export_file("${view}_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), $file_name); } else { $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), "$dirprefix$self->{output}"); } if (!$self->{plsql_pgsql}) { $sql_output .= "DROP MATERIALIZED VIEW $self->{pg_supports_ifexists} $view;\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE MATERIALIZED VIEW $view\n"; if (!$self->{is_mysql} && !$self->{is_mssql}) { $sql_output .= "BUILD $self->{materialized_views}{$view}{build_mode}\n"; $sql_output .= "REFRESH $self->{materialized_views}{$view}{refresh_method} ON $self->{materialized_views}{$view}{refresh_mode}\n"; $sql_output .= "ENABLE QUERY REWRITE" if ($self->{materialized_views}{$view}{rewritable}); $sql_output .= "AS "; } $sql_output .= "$self->{materialized_views}{$view}{text}"; if (!$self->{is_mysql} && !$self->{is_mssql}) { $sql_output .= " USING INDEX" if ($self->{materialized_views}{$view}{no_index}); $sql_output .= " USING NO INDEX" if (!$self->{materialized_views}{$view}{no_index}); } $sql_output .= ";\n\n"; # Set the index definition my ($idx, $fts_idx) = $self->_create_indexes($view, 0, %{$self->{materialized_views}{$view}{indexes}}); $sql_output .= "$idx$fts_idx\n\n"; } else { $self->{materialized_views}{$view}{text} = $self->_format_view($view, $self->{materialized_views}{$view}{text}); if (!$self->{preserve_case}) { $self->{materialized_views}{$view}{text} =~ s/"//gs; } if ($self->{export_schema} && !$self->{schema} && ($view =~ /^([^\.]+)\./) ) { $sql_output .= $self->set_search_path($1) . "\n"; } $self->{materialized_views}{$view}{text} =~ s/^PERFORM/SELECT/; if (!$self->{pg_supports_mview}) { $sql_output .= "DROP VIEW $self->{pg_supports_ifexists} \L$view\E_mview;\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE VIEW \L$view\E_mview AS\n"; if ($self->{is_mssql}) { $self->{materialized_views}{$view}{text} =~ s/^(.*?)\s+AS\s+//is; } $sql_output .= $self->{materialized_views}{$view}{text}; $sql_output .= ";\n\n"; $sql_output .= "SELECT create_materialized_view('\L$view\E','\L$view\E_mview', change with the name of the colum to used for the index);\n\n\n"; if ($self->{force_owner}) { my $owner = $self->{materialized_views}{$view}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER VIEW " . $self->quote_object_name($view . '_mview') . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } else { $sql_output .= "DROP MATERIALIZED VIEW $self->{pg_supports_ifexists} \L$view\E;\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE MATERIALIZED VIEW \L$view\E AS\n"; if ($self->{is_mssql}) { $self->{materialized_views}{$view}{text} =~ s/^(.*?)\s+AS\s+//is; } $sql_output .= $self->{materialized_views}{$view}{text}; if ($self->{materialized_views}{$view}{build_mode} eq 'DEFERRED') { $sql_output .= " WITH NO DATA"; } $sql_output .= ";\n"; # Set the index definition my ($idx, $fts_idx) = $self->_create_indexes($view, 0, %{$self->{materialized_views}{$view}{indexes}}); $sql_output .= "$idx$fts_idx\n\n"; } } if ($self->{force_owner}) { my $owner = $self->{materialized_views}{$view}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER MATERIALIZED VIEW " . $self->quote_object_name($view) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } if ($self->{file_per_table} && !$self->{pg_dsn}) { $self->dump($sql_header . $sql_output, $fhdl); $self->close_export_file($fhdl); $sql_output = ''; } $nothing++; $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_mview, 25, '=', 'materialized views', 'end of output.'), "\n"; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); return; } =head2 export_grant Export Oracle user grants into PostgreSQL compatible SQL statements. =cut sub export_grant { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add users/roles/grants privileges...\n", 1); my $grants = ''; my $users = ''; # Read DML from file if any if ($self->{input_file}) { $self->read_grant_from_file(); } # Do not create privilege defintiion if object type is USER delete $self->{grants} if ($self->{grant_object} && $self->{grant_object} eq 'USER'); # Add privilege definition foreach my $table (sort {"$self->{grants}{$a}{type}.$a" cmp "$self->{grants}{$b}{type}.$b" } keys %{$self->{grants}}) { my $realtable = lc($table); my $obj = $self->{grants}{$table}{type} || 'TABLE'; $obj =~ s/ (PARTITION|SUBPARTITION)//i; if ($self->{export_schema} && $self->{schema}) { $realtable = $self->quote_object_name("$self->{schema}.$table"); } elsif ($self->{preserve_case}) { $realtable = $self->quote_object_name($table); } $grants .= "-- Set priviledge on $self->{grants}{$table}{type} $table\n"; my $ownee = $self->quote_object_name($self->{grants}{$table}{owner}); my $wgrantoption = ''; if ($self->{grants}{$table}{grantable}) { $wgrantoption = ' WITH GRANT OPTION'; } if ($self->{grants}{$table}{type} ne 'PACKAGE BODY') { if ($self->{grants}{$table}{owner}) { if (grep(/^$self->{grants}{$table}{owner}$/, @{$self->{roles}{roles}})) { $grants .= "ALTER $obj $realtable OWNER TO ROLE $ownee;\n"; $obj = '' if (!grep(/^$obj$/, 'FUNCTION', 'PROCEDURE', 'SEQUENCE','SCHEMA','TABLESPACE')); $grants .= "GRANT ALL ON $obj $realtable TO ROLE $ownee$wgrantoption;\n"; } else { $grants .= "ALTER $obj $realtable OWNER TO $ownee;\n"; $obj = '' if (!grep(/^$obj$/, 'FUNCTION', 'PROCEDURE', 'SEQUENCE','SCHEMA','TABLESPACE')); $grants .= "GRANT ALL ON $obj $realtable TO $ownee$wgrantoption;\n"; } if ($realtable =~ /^([^\.]+)\./) { $grants .= "GRANT USAGE ON SCHEMA $1 TO $ownee;\n"; } } if (grep(/^$self->{grants}{$table}{type}$/, 'FUNCTION', 'PROCEDURE', 'SEQUENCE','SCHEMA','TABLESPACE')) { $grants .= "REVOKE ALL ON $self->{grants}{$table}{type} $realtable FROM PUBLIC;\n"; } else { $grants .= "REVOKE ALL ON $realtable FROM PUBLIC;\n"; } } else { $realtable =~ s/^[^\.]+\.//; if ($self->{grants}{$table}{owner}) { if (grep(/^$self->{grants}{$table}{owner}$/, @{$self->{roles}{roles}})) { $grants .= "ALTER SCHEMA $realtable OWNER TO ROLE $ownee;\n"; $grants .= "GRANT EXECUTE ON ALL ROUTINES IN SCHEMA $realtable TO ROLE $ownee$wgrantoption;\n"; } else { $grants .= "ALTER SCHEMA $realtable OWNER TO $ownee;\n"; $grants .= "GRANT EXECUTE ON ALL ROUTINES IN SCHEMA $realtable TO $ownee$wgrantoption;\n"; } } $grants .= "REVOKE ALL ON SCHEMA $realtable FROM PUBLIC;\n"; } foreach my $usr (sort keys %{$self->{grants}{$table}{privilege}}) { my $agrants = ''; foreach my $g (@GRANTS) { $agrants .= "$g," if (grep(/^$g$/i, @{$self->{grants}{$table}{privilege}{$usr}})); } $agrants =~ s/,$//; $usr = $self->quote_object_name($usr); if ($self->{grants}{$table}{type} ne 'PACKAGE BODY') { if (grep(/^$self->{grants}{$table}{type}$/, 'FUNCTION', 'PROCEDURE', 'SEQUENCE','SCHEMA','TABLESPACE', 'TYPE')) { $grants .= "GRANT $agrants ON $obj $realtable TO $usr$wgrantoption;\n"; } else { $grants .= "GRANT $agrants ON $realtable TO $usr$wgrantoption;\n"; } if ($realtable =~ /^([^\.]+)\./) { $grants .= "GRANT USAGE ON SCHEMA $1 TO $usr;\n"; } } else { $realtable =~ s/^[^\.]+\.//; $grants .= "GRANT USAGE ON SCHEMA $realtable TO $usr$wgrantoption;\n"; $grants .= "GRANT EXECUTE ON ALL ROUTINES IN SCHEMA $realtable TO $usr$wgrantoption;\n"; } } $grants .= "\n"; } # Do not create user when privilege on an object type is asked delete $self->{roles} if ($self->{grant_object} && $self->{grant_object} ne 'USER'); foreach my $r (@{$self->{roles}{owner}}, @{$self->{roles}{grantee}}) { my $secret = 'change_my_secret'; if ($self->{gen_user_pwd}) { $secret = &randpattern("CccnCccn"); } $sql_header .= "CREATE " . ($self->{roles}{type}{$r} ||'USER') . " $r"; $sql_header .= " WITH PASSWORD '$secret'" if ($self->{roles}{password_required}{$r} ne 'NO'); # It's difficult to parse all oracle privilege. So if one admin option is set we set all PG admin option. if (grep(/YES|1/, @{$self->{roles}{$r}{admin_option}})) { $sql_header .= " CREATEDB CREATEROLE CREATEUSER INHERIT"; } if ($self->{roles}{type}{$r} eq 'USER') { $sql_header .= " LOGIN"; } if (exists $self->{roles}{role}{$r}) { $users .= " IN ROLE " . join(',', @{$self->{roles}{role}{$r}}); } $sql_header .= ";\n"; } if (!$grants) { $grants = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $sql_output .= "\n" . $grants . "\n" if ($grants); $self->_restore_comments(\$grants); $self->dump($sql_header . $sql_output); return; } =head2 export_sequence Export Oracle sequence into PostgreSQL compatible SQL statements. =cut sub export_sequence { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add sequences definition...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_sequence_from_file(); } my $i = 1; my $num_total_sequence = scalar keys %{$self->{sequences}}; my $count_seq = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_sequence); if ($self->{export_schema} && ($self->{schema} || $self->{pg_schema})) { if ($self->{create_schema}) { $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . ";\n"; } } foreach my $seq (sort keys %{$self->{sequences}}) { if (!$self->{quiet} && !$self->{debug} && ($count_seq % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_sequence, 25, '=', 'sequences', "generating $seq" ), "\r"; } $count_seq++; my $cache = ''; $cache = $self->{sequences}{$seq}->[5] if ($self->{sequences}{$seq}->[5]); my $cycle = ''; $cycle = ' CYCLE' if ($self->{sequences}{$seq}->[6] eq 'Y'); $sql_output .= "DROP SEQUENCE $self->{pg_supports_ifexists} " . $self->quote_object_name($seq) . ";\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE SEQUENCE " . $self->quote_object_name($seq) . " INCREMENT $self->{sequences}{$seq}->[3]"; if ($self->{sequences}{$seq}->[1] eq '' || $self->{sequences}{$seq}->[1] <= (-2**63/2)) { $sql_output .= " NO MINVALUE"; # MSSQL has 32 bit sequences } elsif ($self->{sequences}{$seq}->[1] eq '' || $self->{sequences}{$seq}->[1] <= (-2**32/2)) { $sql_output .= " NO MINVALUE"; } else { $sql_output .= " MINVALUE $self->{sequences}{$seq}->[1]"; } # Max value lower than start value are not allowed if (($self->{sequences}{$seq}->[2] > 0) && ($self->{sequences}{$seq}->[2] < $self->{sequences}{$seq}->[4])) { if (!$cycle) { $self->{sequences}{$seq}->[2] = $self->{sequences}{$seq}->[4]; } else { $self->{sequences}{$seq}->[4] = $self->{sequences}{$seq}->[1]; } } if ($self->{sequences}{$seq}->[2] eq '' || $self->{sequences}{$seq}->[2] >= (2**63/2)-1) { $sql_output .= " NO MAXVALUE"; # MSSQL has 32 bit sequences } elsif ($self->{sequences}{$seq}->[2] eq '' || $self->{sequences}{$seq}->[2] >= (2**32/2)-1) { $sql_output .= " NO MAXVALUE"; } else { $self->{sequences}{$seq}->[2] = 9223372036854775807 if ($self->{sequences}{$seq}->[2] > 9223372036854775807); $sql_output .= " MAXVALUE $self->{sequences}{$seq}->[2]"; } $sql_output .= " START $self->{sequences}{$seq}->[4]"; $sql_output .= " CACHE $cache" if ($cache ne ''); $sql_output .= "$cycle;\n"; if ($self->{force_owner}) { my $owner = $self->{sequences}{$seq}->[7]; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER SEQUENCE " . $self->quote_object_name($seq) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_sequence, 25, '=', 'sequences', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); return; } =head2 export_sequence_values Export Oracle sequence last values into PostgreSQL compatible SQL statements. =cut sub export_sequence_values { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = $self->set_search_path() . "\n"; $self->logit("Add sequences last values...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_sequence_from_file(); } my $i = 1; my $num_total_sequence = scalar keys %{$self->{sequences}}; my $count_seq = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_sequence); foreach my $seq (sort keys %{$self->{sequences}}) { if (!$self->{quiet} && !$self->{debug} && ($count_seq % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_sequence, 25, '=', 'sequences', "generating $seq" ), "\r"; } $count_seq++; $sql_output .= "ALTER SEQUENCE " . $self->quote_object_name($seq) . " START WITH $self->{sequences}{$seq}->[4];\n"; $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_sequence, 25, '=', 'sequences', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); return; } =head2 export_dblink Export Oracle sequence into PostgreSQL compatible SQL statements. =cut sub export_dblink { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add dblink definition...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_dblink_from_file(); } my $i = 1; my $num_total_dblink = scalar keys %{$self->{dblink}}; foreach my $db (sort { $a cmp $b } keys %{$self->{dblink}}) { if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i, $num_total_dblink, 25, '=', 'dblink', "generating $db" ), "\r"; } my $srv_name = $self->quote_object_name($db); $srv_name =~ s/^.*\.//; $sql_output .= "CREATE SERVER $srv_name"; if ($self->{is_mysql}) { $sql_output .= " FOREIGN DATA WRAPPER mysql_fdw OPTIONS (host '$self->{dblink}{$db}{host}'"; $sql_output .= ", port '$self->{dblink}{$db}{port}'" if ($self->{dblink}{$db}{port}); $sql_output .= ");\n"; } elsif ($self->{is_mssql}) { $self->{oracle_dsn} =~ /driver=([^;]+);/; my $driver = $1 || 'msodbcsql18'; $sql_output .= " FOREIGN DATA WRAPPER odbc_fdw OPTIONS (odbc_driver '$driver', odbc_server '$self->{dblink}{$db}{host}'"; $sql_output .= ", odbc_port '$self->{dblink}{$db}{port}'" if ($self->{dblink}{$db}{port}); $sql_output .= ");\n"; } else { $sql_output .= " FOREIGN DATA WRAPPER oracle_fdw OPTIONS (dbserver '$self->{dblink}{$db}{host}');\n"; } if ($self->{dblink}{$db}{password} ne 'NONE') { $self->{dblink}{$db}{password} ||= 'secret'; $self->{dblink}{$db}{password} = ", password '$self->{dblink}{$db}{password}'"; } if ($self->{dblink}{$db}{username}) { my $usr_name = $self->quote_object_name($self->{dblink}{$db}{username}); $usr_name =~ s/^.*\.//; $sql_output .= "CREATE USER MAPPING FOR \"$usr_name\" SERVER $srv_name"; $usr_name = $self->quote_object_name($self->{dblink}{$db}{user}); $usr_name =~ s/^.*\.//; $sql_output .= " OPTIONS (user '$usr_name' $self->{dblink}{$db}{password});\n"; } if ($self->{force_owner}) { my $owner = $self->{dblink}{$db}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER SERVER $srv_name" . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_dblink, 25, '=', 'dblink', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); return; } =head2 export_directory Export Oracle directory into PostgreSQL compatible SQL statements. =cut sub export_directory { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add directory definition...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_directory_from_file(); } my $i = 1; my $num_total_directory = scalar keys %{$self->{directory}}; foreach my $db (sort { $a cmp $b } keys %{$self->{directory}}) { if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i, $num_total_directory, 25, '=', 'directory', "generating $db" ), "\r"; } $sql_output .= "INSERT INTO external_file.directories (directory_name,directory_path) VALUES ('$db', '$self->{directory}{$db}{path}');\n"; foreach my $owner (keys %{$self->{directory}{$db}{grantee}}) { my $write = 'false'; $write = 'true' if ($self->{directory}{$db}{grantee}{$owner} =~ /write/i); $sql_output .= "INSERT INTO external_file.directory_roles(directory_name,directory_role,directory_read,directory_write) VALUES ('$db','" . $self->quote_object_name($owner) . "', true, $write);\n"; } $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_directory, 25, '=', 'directory', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); return; } sub _replace_sql_type { my ($self, $str) = @_; if ($self->{is_mysql}) { $str = Ora2Pg::MySQL::replace_sql_type($self, $str); } elsif ($self->{is_mssql}) { $str = Ora2Pg::MSSQL::replace_sql_type($self, $str); } else { $str = Ora2Pg::PLSQL::replace_sql_type($self, $str); } return $str; } =head2 export_trigger Export Oracle trigger into PostgreSQL compatible SQL statements. =cut sub export_trigger { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add triggers definition...\n", 1); $self->dump($sql_header); # Read DML from file if any if ($self->{input_file}) { $self->read_trigger_from_file(); } my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $nothing = 0; my $i = 1; my $num_total_trigger = $#{$self->{triggers}} + 1; my $count_trig = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_trigger); foreach my $trig (sort {$a->[0] cmp $b->[0]} @{$self->{triggers}}) { if (!$self->{quiet} && !$self->{debug} && ($count_trig % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_trigger, 25, '=', 'triggers', "generating $trig->[0]" ), "\r"; } $count_trig++; my $fhdl = undef; if ($self->{file_per_function}) { my $schm = ''; $schm = $trig->[8] . '-' if ($trig->[8] && $self->{export_schema} && !$self->{schema}); my $f = "$dirprefix$schm$trig->[0]_$self->{output}"; $f = "$schm$trig->[0]_$self->{output}" if ($self->{psql_relative_path}); $f =~ s/\.(?:gz|bz2)$//i; $self->dump("\\i$self->{psql_relative_path} $f\n"); $self->logit("Dumping to one file per trigger : $schm$trig->[0]_$self->{output}\n", 1); $fhdl = $self->open_export_file("$schm$trig->[0]_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($trig->[0]), "$dirprefix$schm$trig->[0]_$self->{output}"); } else { $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($trig->[0]), "$dirprefix$self->{output}"); } $trig->[1] =~ s/\s*EACH ROW//is; chomp($trig->[4]); $trig->[4] =~ s/([^\*])[;\/]$/$1/; # reordering of event when there is a OF keyword to specify columns $trig->[2] =~ s/(\s+OR\s+UPDATE)(\s+.*)(\s+OF\s+)/$2$1$3/i; $self->logit("\tDumping trigger $trig->[0] defined on table $trig->[3]...\n", 1); my $tbname = $self->get_replaced_tbname($trig->[3]); # Store current trigger table name for possible use in outer join translation $self->{current_trigger_table} = $trig->[3]; # Replace column name in function code if (exists $self->{replaced_cols}{"\L$trig->[3]\E"}) { foreach my $coln (sort keys %{$self->{replaced_cols}{"\L$trig->[3]\E"}}) { $self->logit("\tReplacing column \L$coln\E as " . $self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"} . "...\n", 1); my $cname = $self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"}; $cname = $self->quote_object_name($cname); $trig->[4] =~ s/(OLD|NEW)\.$coln\b/$1\.$cname/igs; $trig->[6] =~ s/\b$coln\b/$self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"}/is; } } # Extract columns specified in the UPDATE OF ... ON clause my $cols = ''; if ($trig->[2] =~ /UPDATE/ && $trig->[6] =~ /UPDATE\s+OF\s+(.*?)\s+ON/i) { my @defs = split(/\s*,\s*/, $1); $cols = ' OF '; foreach my $c (@defs) { $cols .= $self->quote_object_name($c) . ','; } $cols =~ s/,$//; } if ($self->{export_schema} && !$self->{schema}) { $sql_output .= $self->set_search_path($trig->[8]) . "\n"; } # Check if it's like a pg rule $self->_remove_comments(\$trig->[4]); if (!$self->{pg_supports_insteadof} && $trig->[1] =~ /INSTEAD OF/) { if ($self->{plsql_pgsql}) { $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); $self->_replace_declare_var(\$trig->[4]); } $sql_output .= "CREATE$self->{create_or_replace} RULE " . $self->quote_object_name($trig->[0]) . " AS\n\tON " . $self->quote_object_name($trig->[2]) . " TO " . $self->quote_object_name($tbname) . "\n\tDO INSTEAD\n(\n\t$trig->[4]\n);\n\n"; if ($self->{force_owner}) { my $owner = $trig->[8]; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER RULE " . $self->quote_object_name($trig->[0]) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } else { # Replace direct call of a stored procedure in triggers if ($trig->[7] eq 'CALL') { if ($self->{plsql_pgsql}) { $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); $self->_replace_declare_var(\$trig->[4]); } $trig->[4] = "BEGIN\nPERFORM $trig->[4];\nEND;"; } else { my $ret_kind = 'RETURN NEW;'; if (uc($trig->[2]) eq 'DELETE') { $ret_kind = 'RETURN OLD;'; } elsif (uc($trig->[2]) =~ /DELETE/) { $ret_kind = "IF TG_OP = 'DELETE' THEN\n\tRETURN OLD;\nELSE\n\tRETURN NEW;\nEND IF;\n"; } if ($self->{plsql_pgsql}) { # Add a semi colon if none if ($trig->[4] !~ /\bBEGIN\b/i) { chomp($trig->[4]); $trig->[4] .= ';' if ($trig->[4] !~ /;\s*$/s); $trig->[4] = "BEGIN\n$trig->[4]\n$ret_kind\nEND;"; } $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); $self->_replace_declare_var(\$trig->[4]); # When an exception statement is used enclosed everything # in a block before returning NEW if ($trig->[4] =~ /EXCEPTION(.*?)\b(END[;]*)[\s\/]*$/is) { $trig->[4] =~ s/^\s*BEGIN/BEGIN\n BEGIN/ism; $trig->[4] =~ s/\b(END[;]*)[\s\/]*$/ END;\n$1/is; } # Add return statement. $trig->[4] =~ s/(?:$ret_kind\s+)?\b(END[;]*)(\s*\%ORA2PG_COMMENT\d+\%\s*)?[\s\/]*$/$ret_kind\n$1$2/igs; # Look at function header to convert sql type my @parts = split(/BEGIN/i, $trig->[4]); if ($#parts > 0) { $parts[0] = $self->_replace_sql_type($parts[0]); } $trig->[4] = join('BEGIN', @parts); $trig->[4] =~ s/\bRETURN\s*;/$ret_kind/igs; } } $sql_output .= "DROP TRIGGER $self->{pg_supports_ifexists} " . $self->quote_object_name($trig->[0]) . " ON " . $self->quote_object_name($tbname) . " CASCADE;\n"; my $security = ''; my $revoke = ''; my $trig_fctname = $self->quote_object_name("trigger_fct_\L$trig->[0]\E"); if ($self->{export_schema} && ($self->{pg_schema} || $self->{schema} || $trig->[8])) { $trig_fctname = $self->quote_object_name($self->{pg_schema} || $self->{schema} || $trig->[8]) . ".$trig_fctname"; } if ($self->{security}{"\U$trig->[0]\E"}{security} eq 'DEFINER') { $security = " SECURITY DEFINER"; $revoke = "-- REVOKE ALL ON FUNCTION $trig_fctname() FROM PUBLIC;\n"; } $security = " SECURITY INVOKER" if ($self->{force_security_invoker}); $trig->[4] =~ s/CREATE TRIGGER (.*?)\sAS\s+//s; if ($self->{pg_supports_when} && $trig->[5]) { if (!$self->{preserve_case}) { $trig->[4] =~ s/"([^"]+)"/\L$1\E/gs; $trig->[4] =~ s/ALTER TRIGGER\s+[^\s]+\s+ENABLE(;)?//; } $sql_output .= "CREATE$self->{create_or_replace} FUNCTION $trig_fctname() RETURNS trigger AS \$BODY\$\n$trig->[4]\n\$BODY\$\n LANGUAGE 'plpgsql'$security;\n$revoke\n"; if ($self->{force_owner}) { my $owner = $trig->[8]; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER FUNCTION $trig_fctname() OWNER TO " . $self->quote_object_name($owner) . ";\n\n"; } $self->_remove_comments(\$trig->[6]); $trig->[6] =~ s/\n+$//s; $trig->[6] =~ s/^[^\.\s]+\.//; if (!$self->{preserve_case}) { $trig->[6] =~ s/"([^"]+)"/\L$1\E/gs; } chomp($trig->[6]); # Remove referencing clause, not supported by PostgreSQL < 10 if ($self->{pg_version} < 10) { $trig->[6] =~ s/REFERENCING\s+(.*?)(FOR\s+EACH\s+)/$2/is; } $trig->[6] =~ s/REFERENCING\s+(NEW|OLD)\s+AS\s+(NEW|OLD)\s+(NEW|OLD)\s+AS\s+(NEW|OLD)//gsi; $trig->[6] =~ s/^\s*["]*(?:$trig->[0])["]*//is; $trig->[6] =~ s/\s+ON\s+([^"\s]+)\s+/" ON " . $self->quote_object_name($1) . " "/ies; $sql_output .= "DROP TRIGGER $self->{pg_supports_ifexists} " . $self->quote_object_name($trig->[0]) . " ON " . $self->quote_object_name($1) . ";\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE TRIGGER " . $self->quote_object_name($trig->[0]) . "$trig->[6]\n"; if ($trig->[5]) { $self->_remove_comments(\$trig->[5]); $trig->[5] =~ s/"([^"]+)"/\L$1\E/gs if (!$self->{preserve_case}); if ($self->{plsql_pgsql}) { $trig->[5] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[5]); $self->_replace_declare_var(\$trig->[5]); } $sql_output .= "\tWHEN ($trig->[5])\n"; } if ($trig->[6] =~ /REFERENCING/) { $trig->[6] =~ s/REFERENCING\s+(NEW|OLD)\s+AS\s+(NEW|OLD)\s+(NEW|OLD)\s+AS\s+(NEW|OLD)//gsi; $sql_output .= "$trig->[6] "; } $sql_output .= "\tEXECUTE PROCEDURE $trig_fctname();\n\n"; } else { if (!$self->{preserve_case}) { $trig->[4] =~ s/"([^"]+)"/\L$1\E/gs; $trig->[4] =~ s/ALTER TRIGGER\s+[^\s]+\s+ENABLE(;)?//; } $sql_output .= "CREATE$self->{create_or_replace} FUNCTION $trig_fctname() RETURNS trigger AS \$BODY\$\n$trig->[4]\n\$BODY\$\n LANGUAGE 'plpgsql'$security;\n$revoke\n"; if ($self->{force_owner}) { my $owner = $trig->[8]; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER FUNCTION $trig_fctname() OWNER TO " . $self->quote_object_name($owner) . ";\n\n"; } $sql_output .= "DROP TRIGGER $self->{pg_supports_ifexists} " . $self->quote_object_name($trig->[0]) . " ON " . $self->quote_object_name($tbname) . ";\n" if ($self->{drop_if_exists}); $sql_output .= "CREATE TRIGGER " . $self->quote_object_name($trig->[0]) . "\n\t"; my $statement = 0; $statement = 1 if ($trig->[1] =~ s/ STATEMENT//); $sql_output .= "$trig->[1] $trig->[2]$cols ON " . $self->quote_object_name($tbname) . " "; if ($trig->[6] =~ s/.*(REFERENCING\s+.*)/$1/is) { $trig->[6] =~ s/REFERENCING\s+(NEW|OLD)\s+AS\s+(NEW|OLD)(\s+(NEW|OLD)\s+AS\s+(NEW|OLD))?//gsi; $trig->[6] =~ s/\s+FOR EACH ROW//gsi; $sql_output .= "$trig->[6] "; } if ($self->{is_mssql}) { my $reftb = "REFERENCING OLD TABLE AS Deleted NEW TABLE AS Inserted"; $reftb =~ s/OLD TABLE AS Deleted // if ($trig->[2] eq 'INSERT'); $reftb =~ s/NEW TABLE AS Inserted // if ($trig->[2] eq 'DELETE'); $sql_output .= "$reftb FOR EACH STATEMENT\n" if ($trig->[1] !~ /INSTEAD OF/is); } elsif ($statement) { $sql_output .= "FOR EACH STATEMENT\n" if ($trig->[1] !~ /INSTEAD OF/is); } else { $sql_output .= "FOR EACH ROW\n" if ($trig->[1] !~ /INSTEAD OF/is); } $sql_output .= "\tEXECUTE PROCEDURE $trig_fctname();\n\n"; } } $self->_restore_comments(\$sql_output); if ($self->{file_per_function}) { $self->dump($sql_header . $sql_output, $fhdl); $self->close_export_file($fhdl); $sql_output = ''; } $nothing++; $i++; } delete $self->{current_trigger_table}; if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_trigger, 25, '=', 'triggers', 'end of output.'), "\n"; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); return; } =head2 parallelize_statements Parallelize SQL statements to import into PostgreSQL. =cut sub parallelize_statements { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Parse SQL orders to load...\n", 1); my $nothing = 0; #--------------------------------------------------------- # Load a file containing SQL code to load into PostgreSQL #--------------------------------------------------------- my %comments = (); my @settings = (); if ($self->{input_file}) { $self->{functions} = (); $self->logit("Reading input SQL orders from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); # remove comments only, text constants are preserved $self->_remove_comments(\$content, 1); $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; my $query = 1; foreach my $l (split(/\n/, $content)) { chomp($l); next if ($l =~ /^\s*$/); # do not parse interactive or session command next if ($l =~ /^(\\set|\\pset|\\i)/is); # Put setting change in header to apply them on all parallel session # This will help to set a special search_path or encoding if ($l =~ /^SET\s+/i) { push(@settings, $l); next; } if ($old_line) { $l = $old_line .= ' ' . $l; $old_line = ''; } if ($l =~ /;\s*$/) { $self->{queries}{$query} .= "$l\n"; $query++; } else { $self->{queries}{$query} .= "$l\n"; } } } else { $self->logit("No input file, aborting...\n", 0, 1); } #-------------------------------------------------------- my $total_queries = scalar keys %{$self->{queries}}; $self->{child_count} = 0; foreach my $q (sort {$a <=> $b} keys %{$self->{queries}}) { chomp($self->{queries}{$q}); next if (!$self->{queries}{$q}); if ($self->{jobs} > 1) { while ($self->{child_count} >= $self->{jobs}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->_pload_to_pg($q, $self->{queries}{$q}, @settings); }; $self->{child_count}++; } else { $self->_pload_to_pg($q, $self->{queries}{$q}, @settings); } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($q, $total_queries, 25, '=', 'queries', "dispatching query #$q" ), "\r"; } $nothing++; } $self->{queries} = (); if (!$total_queries) { $self->logit("No query to load...\n", 0); } else { # Wait for all child end while ($self->{child_count} > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } if (!$self->{quiet} && !$self->{debug}) { print STDERR "\n"; } } return; } =head2 translate_query Translate Oracle's queries into PostgreSQL compatible statement. =cut sub translate_query { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Parse queries definition...\n", 1); $self->dump($sql_header); my $nothing = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); #--------------------------------------------------------- # Code to use to find queries parser issues, it load a file # containing the untouched SQL code from Oracle queries #--------------------------------------------------------- if ($self->{input_file}) { $self->{functions} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); $self->_remove_comments(\$content); my $query = 1; foreach my $l (split(/(?:^\/$|;\s*$)/m, $content)) { chomp($l); next if ($l =~ /^\s*$/s); $self->{queries}{$query}{code} = "$l\n"; $query++; } $content = ''; foreach my $q (keys %{$self->{queries}}) { $self->_restore_comments(\$self->{queries}{$q}{code}); } } foreach my $q (sort { $a <=> $b } keys %{$self->{queries}}) { if ($self->{queries}{$q}{code} !~ /(SELECT|UPDATE|DELETE|INSERT|DROP|TRUNCATE|CREATE(?:UNIQUE)? INDEX|ALTER)/is) { $self->{queries}{$q}{to_be_parsed} = 0; } else { $self->{queries}{$q}{to_be_parsed} = 1; } } #-------------------------------------------------------- my $total_size = 0; my $cost_value = 0; foreach my $q (sort {$a <=> $b} keys %{$self->{queries}}) { $total_size += length($self->{queries}{$q}{code}); $self->logit("Dumping query $q...\n", 1); if ($self->{queries}{$q}{to_be_parsed}) { if ($self->{plsql_pgsql}) { $self->_remove_comments(\$self->{queries}{$q}{code}); if (!$self->{preserve_case}) { $self->{queries}{$q}{code} =~ s/"//gs; } my $sql_q = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{queries}{$q}{code}); my $estimate = ''; if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_q, 'QUERY'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'QUERY'}; $cost_value += $cost; $estimate = "\n-- Estimed cost of query [ $q ]: " . sprintf("%2.2f", $cost); } $self->_restore_comments(\$sql_q); $sql_output .= $sql_q; $sql_output .= ';' if ($sql_q !~ /;\s*$/); $sql_output .= $estimate; } else { $sql_output .= $self->{queries}{$q}{code}; } } else { $sql_output .= $self->{queries}{$q}{code}; $sql_output .= ';' if ($self->{queries}{$q}{code} !~ /;\s*$/); } $sql_output .= "\n"; $nothing++; } if ($self->{estimate_cost}) { $cost_value = sprintf("%2.2f", $cost_value); my @infos = ( "Total number of queries: ".(scalar keys %{$self->{queries}}).".", "Total size of queries code: $total_size bytes.", "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." ); $self->logit(join("\n", @infos) . "\n", 1); map { s/^/-- /; } @infos; $sql_output .= join("\n", @infos); } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); $self->{queries} = (); return; } =head2 translate_script Translate Oracle's SQL script into PostgreSQL compatible scripts. =cut sub translate_script { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Parse whole script...\n", 1); $self->dump($sql_header); my $nothing = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); #--------------------------------------------------------- # Code to use to find queries parser issues, it load a file # containing the untouched SQL code from Oracle queries #--------------------------------------------------------- if ($self->{input_file}) { $self->{functions} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); $self->{script}{code} = $self->read_input_file($self->{input_file}); #$self->_remove_comments(\$self->{script}{code}); #$self->_restore_comments(\$self->{script}{code}); chomp($self->{script}{code}); if (length($self->{script}{code}) > 0) { $nothing = 1; } } else { $self->logit("FATAL: the SCRIPT action need an input file, see -i option\n", 0, 1); } #-------------------------------------------------------- my $cost_value = 0; my $total_size = length($self->{script}{code}) || 0; $self->logit("Dumping script...\n", 1); $self->_remove_comments(\$self->{script}{code}); if (!$self->{preserve_case}) { $self->{script}{code} =~ s/"//gs; } my $sql_q = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{script}{code}); my $estimate = ''; if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_q, 'SCRIPT'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'QUERY'}; $cost_value += $cost; $estimate = "\n-- Estimed cost of script [ $self->{input_file} ]: " . sprintf("%2.2f", $cost); } $self->_restore_comments(\$sql_q); # replace script parameters $sql_q =~ s/'\&(\d+)'/:'param$1'/gs; $sql_q =~ s/\&(\d+)/:param$1/gs; $sql_output .= $sql_q; $sql_output .= ';' if ($sql_q !~ /;\s*$/); $sql_output .= $estimate; if ($self->{estimate_cost}) { $cost_value = sprintf("%2.2f", $cost_value); my @infos = ("Total size of queries code: $total_size bytes.", "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." ); $self->logit(join("\n", @infos) . "\n", 1); map { s/^/-- /; } @infos; $sql_output .= join("\n", @infos); } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); $self->{script} = (); return; } =head2 export_function Export Oracle functions into PostgreSQL compatible statement. =cut sub export_function { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; use constant SQL_DATATYPE => 2; $self->logit("Add functions definition...\n", 1); $self->dump($sql_header); my $nothing = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); #--------------------------------------------------------- # Code to use to find function parser issues, it load a file # containing the untouched PL/SQL code from Oracle Function #--------------------------------------------------------- if ($self->{input_file}) { $self->{functions} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); $self->_remove_comments(\$content); my @allfct = split(/\n/, $content); my $fcnm = ''; my $old_line = ''; my $language = ''; foreach my $l (@allfct) { chomp($l); $l =~ s/\r//g; next if ($l =~ /^\s*$/); if ($old_line) { $l = $old_line .= ' ' . $l; $old_line = ''; } if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*$/i) { $old_line = $l; next; } if ($l =~ /^\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)$/i) { $old_line = $l; next; } if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)\s*$/i) { $old_line = $l; next; } $l =~ s/^\s*CREATE (?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)/$1/i; $l =~ s/^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; if ($l =~ /^(FUNCTION|PROCEDURE)\s+([^\s\(]+)/i) { $fcnm = $2; $fcnm =~ s/"//g; } next if (!$fcnm); if ($l =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { $language = $1; } $self->{functions}{$fcnm}{text} .= "$l\n"; if (!$language) { if ($l =~ /^END\s+$fcnm(_atx)?\s*;/i) { $fcnm = ''; } } else { if ($l =~ /;/i) { $fcnm = ''; $language = ''; } } } # Get all metadata from all functions when we are # reading a file, otherwise it has already been done foreach my $fct (sort keys %{$self->{functions}}) { $self->{functions}{$fct}{text} =~ s/(.*?\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; my %fct_detail = $self->_lookup_function($self->{functions}{$fct}{text}, ($self->{is_mysql}) ? $fct : undef); if (!exists $fct_detail{name}) { delete $self->{functions}{$fct}; next; } $self->{functions}{$fct}{type} = uc($fct_detail{type}); delete $fct_detail{code}; delete $fct_detail{before}; my $sch = 'unknown'; my $fname = $fct; if ($fname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { $sch = $1; } $fname =~ s/"//g; %{$self->{function_metadata}{$sch}{'none'}{$fname}{metadata}} = %fct_detail; $self->_restore_comments(\$self->{functions}{$fct}{text}); } } #-------------------------------------------------------- my $total_size = 0; my $cost_value = 0; my $num_total_function = scalar keys %{$self->{functions}}; my $fct_cost = ''; my $parallel_fct_count = 0; unlink($dirprefix . 'temp_cost_file.dat') if ($self->{parallel_tables} > 1 && $self->{estimate_cost}); my $t0 = Benchmark->new; # Group functions by chunk in multiprocess mode my $num_chunk = $self->{jobs} || 1; my @fct_group = (); my $i = 0; foreach my $key ( sort keys %{$self->{functions}} ) { if ($self->{print_dependencies} && $self->{plsql_pgsql} && !$self->{no_function_metadata}) { my $plsql_code = $self->{functions}{$key}{text}; $plsql_code =~ s/FUNCTION $key//i; $self->_remove_comments(\$plsql_code); # look for other routines call in the stored function foreach my $sch (sort keys %{ $self->{function_metadata} }) { foreach my $pkg_name (sort keys %{ $self->{function_metadata}{$sch} }) { foreach my $fname (sort keys %{ $self->{function_metadata}{$sch}{$pkg_name} }) { next if ($key =~ /^$fname$/i || $key =~ /^.*\.$fname$/i); if ($plsql_code =~ /\b$fname\b/is) { push(@{ $self->{object_dependencies}{uc("$self->{functions}{$key}{owner}.$key")}{routines} }, uc("$sch.$fname")); } } } } # Look for merge/insert/update/delete @{ $self->{object_dependencies}{uc("$self->{functions}{$key}{owner}.$key")}{merge} } = $plsql_code =~ /\bMERGE\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$self->{functions}{$key}{owner}.$key")}{insert} } = $plsql_code =~ /\bINSERT\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$self->{functions}{$key}{owner}.$key")}{update} } = $plsql_code =~ /(?:(?!FOR).)*?\s*\bUPDATE\s+([^\s;,]+)\s+/igs; @{ $self->{object_dependencies}{uc("$self->{functions}{$key}{owner}.$key")}{delete} } = $plsql_code =~ /\b(?:DELETE\s+FROM|TRUNCATE\s+TABLE)\s+([^\s;,]+)\s+/igs; } $fct_group[$i++]{$key} = $self->{functions}{$key}; $i = 0 if ($i == $num_chunk); } my $num_cur_fct = 0; for ($i = 0; $i <= $#fct_group; $i++) { if ($self->{jobs} > 1) { $self->logit("Creating a new process to translate functions...\n", 1); spawn sub { $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); }; $parallel_fct_count++; } else { my ($code, $lsize, $lcost) = $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); $sql_output .= $code; $total_size += $lsize; $cost_value += $lcost; } $num_cur_fct += scalar keys %{$fct_group[$i]}; $nothing++; } # Wait for all oracle connection terminaison if ($self->{jobs} > 1) { while ($parallel_fct_count) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_fct_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } if ($self->{estimate_cost}) { my $tfh = $self->read_export_file($dirprefix . 'temp_cost_file.dat'); flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; while (my $l = <$tfh>) { chomp($l); my ($fname, $fsize, $fcost) = split(/:/, $l); $total_size += $fsize; $cost_value += $fcost; } $self->close_export_file($tfh, 1); unlink($dirprefix . 'temp_cost_file.dat'); } } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($num_cur_fct, $num_total_function, 25, '=', 'functions', 'end of functions export.'), "\n"; } if ($self->{estimate_cost}) { my @infos = ( "Total number of functions: ".(scalar keys %{$self->{functions}}).".", "Total size of function code: $total_size bytes.", "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." ); $self->logit(join("\n", @infos) . "\n", 1); map { s/^/-- /; } @infos; $sql_output .= "\n" . join("\n", @infos); $sql_output .= "\n" . $fct_cost; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); if (scalar keys %{ $self->{object_dependencies} } > 0) { my $sp_tree = "object_type;object_name;routines_called;insert;update;delete;merge\n"; foreach my $caller ( sort keys %{ $self->{object_dependencies} } ) { $sp_tree .= "FUNCTION;$caller"; $sp_tree .= ";"; foreach my $sp (@{ $self->{object_dependencies}{$caller}{routines} }) { my $star = ($#{ $self->{object_dependencies}{$sp}{routines} } >= 0) ? '*' : ''; $sp_tree .= "$sp$star,"; } $sp_tree =~ s/,$//; $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{insert} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{update} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{delete} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{merge} }); $sp_tree .= "\n"; } my $fhdl = $self->open_export_file("functions_dependencies.csv"); $self->dump($sp_tree, $fhdl); $self->close_export_file($fhdl); } $self->{functions} = (); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Total time to translate all functions with $num_chunk process: " . timestr($td) . "\n", 1); return; } =head2 export_procedure Export Oracle procedures into PostgreSQL compatible statement. =cut sub export_procedure { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; use constant SQL_DATATYPE => 2; $self->logit("Add procedures definition...\n", 1); my $nothing = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); $self->dump($sql_header); #--------------------------------------------------------- # Code to use to find procedure parser issues, it load a file # containing the untouched PL/SQL code from Oracle Procedure #--------------------------------------------------------- if ($self->{input_file}) { $self->{procedures} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); $self->_remove_comments(\$content); my @allfct = split(/\n/, $content); my $fcnm = ''; my $old_line = ''; my $language = ''; my $first_comment = ''; foreach my $l (@allfct) { $l =~ s/\r//g; next if ($l =~ /^\/$/); next if ($l =~ /^\s*$/); if ($old_line) { $l = $old_line .= ' ' . $l; $old_line = ''; } $comment .= $l if ($l =~ /^\%ORA2PG_COMMENT\d+\%$/); if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*$/i) { $old_line = $comment . $l; $comment = ''; next; } if ($l =~ /^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)$/i) { $old_line = $comment . $l; $comment = ''; next; } if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)\s*$/i) { $old_line = $comment . $l; $comment = ''; next; } $l =~ s/^\s*CREATE (?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; $l =~ s/^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; if ($l =~ /^(FUNCTION|PROCEDURE)\s+([^\s\(]+)/i) { $fcnm = $2; $fcnm =~ s/"//g; } next if (!$fcnm); if ($l =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { $language = $1; } if ($comment) { $self->{procedures}{$fcnm}{text} .= "$comment"; $comment = ''; } $self->{procedures}{$fcnm}{text} .= "$l\n"; if (!$language) { if ($l =~ /^END\s+$fcnm(_atx)?\s*;/i) { $fcnm = ''; } } else { if ($l =~ /;/i) { $fcnm = ''; $language = ''; } } } # Get all metadata from all procedures when we are # reading a file, otherwise it has already been done foreach my $fct (sort keys %{$self->{procedures}}) { $self->{procedures}{$fct}{text} =~ s/(.*?\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; my %fct_detail = $self->_lookup_function($self->{procedures}{$fct}{text}, ($self->{is_mysql}) ? $fct : undef); if (!exists $fct_detail{name}) { delete $self->{procedures}{$fct}; next; } $self->{procedures}{$fct}{type} = $fct_detail{type}; delete $fct_detail{code}; delete $fct_detail{before}; my $sch = 'unknown'; my $fname = $fct; if ($fname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { $sch = $1; } $fname =~ s/"//g; %{$self->{function_metadata}{$sch}{'none'}{$fname}{metadata}} = %fct_detail; $self->_restore_comments(\$self->{procedures}{$fct}{text}); } } #-------------------------------------------------------- my $total_size = 0; my $cost_value = 0; my $num_total_function = scalar keys %{$self->{procedures}}; my $fct_cost = ''; my $parallel_fct_count = 0; unlink($dirprefix . 'temp_cost_file.dat') if ($self->{parallel_tables} > 1 && $self->{estimate_cost}); my $t0 = Benchmark->new; # Group functions by chunk in multiprocess mode my $num_chunk = $self->{jobs} || 1; my @fct_group = (); my $i = 0; foreach my $key (sort keys %{$self->{procedures}} ) { if ($self->{print_dependencies} && $self->{plsql_pgsql} && !$self->{no_function_metadata}) { my $plsql_code = $self->{procedures}{$key}{text}; $plsql_code =~ s/FUNCTION $key//i; $self->_remove_comments(\$plsql_code); # look for other routines call in the stored procedure foreach my $sch (sort keys %{ $self->{function_metadata} }) { foreach my $pkg_name (sort keys %{ $self->{function_metadata}{$sch} }) { foreach my $fname (sort keys %{ $self->{function_metadata}{$sch}{$pkg_name} }) { next if ($key =~ /^$fname$/i || $key =~ /^.*\.$fname$/i); if ($plsql_code =~ /\b$fname\b/is) { push(@{ $self->{object_dependencies}{uc("$self->{procedures}{$key}{owner}.$key")}{routines} }, uc("$sch.$fname")); } } } } # Look for merge/insert/update/delete @{ $self->{object_dependencies}{uc("$self->{procedures}{$key}{owner}.$key")}{merge} } = $plsql_code =~ /\bMERGE\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$self->{procedures}{$key}{owner}.$key")}{insert} } = $plsql_code =~ /\bINSERT\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$self->{procedures}{$key}{owner}.$key")}{update} } = $plsql_code =~ /(?:(?!FOR).)*?\s*\bUPDATE\s+([^\s;,]+)\s+/igs; @{ $self->{object_dependencies}{uc("$self->{procedures}{$key}{owner}.$key")}{delete} } = $plsql_code =~ /\b(?:DELETE\s+FROM|TRUNCATE\s+TABLE)\s+([^\s;,]+)\s+/igs; } $fct_group[$i++]{$key} = $self->{procedures}{$key}; $i = 0 if ($i == $num_chunk); } my $num_cur_fct = 0; for ($i = 0; $i <= $#fct_group; $i++) { if ($self->{jobs} > 1) { $self->logit("Creating a new process to translate procedures...\n", 1); spawn sub { $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); }; $parallel_fct_count++; } else { my ($code, $lsize, $lcost) = $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); $sql_output .= $code; $total_size += $lsize;; $cost_value += $lcost; } $num_cur_fct += scalar keys %{$fct_group[$i]}; $nothing++; } # Wait for all oracle connection terminaison if ($self->{jobs} > 1) { while ($parallel_fct_count) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_fct_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } if ($self->{estimate_cost}) { my $tfh = $self->read_export_file($dirprefix . 'temp_cost_file.dat'); flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; if (defined $tfh) { while (my $l = <$tfh>) { chomp($l); my ($fname, $fsize, $fcost) = split(/:/, $l); $total_size += $fsize; $cost_value += $fcost; } $self->close_export_file($tfh, 1); } unlink($dirprefix . 'temp_cost_file.dat'); } } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($num_cur_fct, $num_total_function, 25, '=', 'procedures', 'end of procedures export.'), "\n"; } if ($self->{estimate_cost}) { my @infos = ( "Total number of procedures: ".(scalar keys %{$self->{procedures}}).".", "Total size of procedures code: $total_size bytes.", "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." ); $self->logit(join("\n", @infos) . "\n", 1); map { s/^/-- /; } @infos; $sql_output .= "\n" . join("\n", @infos); $sql_output .= "\n" . $fct_cost; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); if (scalar keys %{ $self->{object_dependencies} } > 0) { my $sp_tree = "object_type;object_name;routines_called;insert;update;delete;merge\n"; foreach my $caller ( sort keys %{ $self->{object_dependencies} } ) { $sp_tree .= "PROCEDURE;$caller"; $sp_tree .= ";"; foreach my $sp (@{ $self->{object_dependencies}{$caller}{routines} }) { my $star = ($#{ $self->{object_dependencies}{$sp}{routines} } >= 0) ? '*' : ''; $sp_tree .= "$sp$star,"; } $sp_tree =~ s/,$//; $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{insert} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{update} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{delete} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{merge} }); $sp_tree .= "\n"; } my $fhdl = $self->open_export_file("procedures_dependencies.csv"); $self->dump($sp_tree, $fhdl); $self->close_export_file($fhdl); } $self->{procedures} = (); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Total time to translate all functions with $num_chunk process: " . timestr($td) . "\n", 1); return; } =head2 export_package Export Oracle package into PostgreSQL compatible statement. =cut sub export_package { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->{current_package} = ''; $self->logit("Add packages definition...\n", 1); my $nothing = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); $self->dump($sql_header); #--------------------------------------------------------- # Code to use to find package parser bugs, it load a file # containing the untouched PL/SQL code from Oracle Package #--------------------------------------------------------- if ($self->{input_file}) { $self->{plsql_pgsql} = 1; $self->{packages} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); my $pknm = ''; my $before = ''; my $old_line = ''; my $skip_pkg_header = 0; $self->_remove_comments(\$content); # Normalise start of package declaration $content =~ s/CREATE(?:\s+OR\s+REPLACE)?(?:\s+EDITIONABLE|\s+NONEDITIONABLE)?\s+PACKAGE\s+/CREATE OR REPLACE PACKAGE /igs; # Preserve header $content =~ s/^(.*?)(CREATE OR REPLACE PACKAGE)/$2/s; my $start = $1 || ''; my @pkg_content = split(/CREATE OR REPLACE PACKAGE\s+/is, $content); for (my $i = 0; $i <= $#pkg_content; $i++) { # package declaration if ($pkg_content[$i] !~ /^BODY\s+/is) { if ($pkg_content[$i] =~ /^([^\s]+)/is) { my $pname = lc($1); $pname =~ s/"//g; $pname =~ s/^[^\.]+\.//g; $self->{packages}{$pname}{desc} = 'CREATE OR REPLACE PACKAGE ' . $pkg_content[$i]; $self->{packages}{$pname}{text} = $start if ($start); $start = ''; } } else { if ($pkg_content[$i] =~ /^BODY\s+([^\s]+)\s+/is) { my $pname = lc($1); $pname =~ s/"//g; $pname =~ s/^[^\.]+\.//g; $self->{packages}{$pname}{text} .= 'CREATE OR REPLACE PACKAGE ' . $pkg_content[$i]; } } } @pkg_content = (); foreach my $pkg (sort keys %{$self->{packages}}) { my $tmp_txt = ''; if (exists $self->{packages}{$pkg}{desc}) { # Move comments at end of package declaration before package definition while ($self->{packages}{$pkg}{desc} =~ s/(\%ORA2PG_COMMENT\d+\%\s*)$//) { $self->{packages}{$pkg}{text} = $1 . $self->{packages}{$pkg}{text}; } } # Get all metadata from all procedures/function when we are # reading from a file, otherwise it has already been done $tmp_txt = $self->{packages}{$pkg}{text}; $tmp_txt =~ s/^.*CREATE OR REPLACE PACKAGE\s+/CREATE OR REPLACE PACKAGE /s; my %infos = $self->_lookup_package($tmp_txt); my $sch = 'unknown'; my $pname = $pkg; if ($pname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { $sch = $1; } foreach my $f (sort keys %infos) { next if (!$f); my $name = lc($f); delete $infos{$f}{code}; delete $infos{$f}{before}; $pname =~ s/"//g; $name =~ s/"//g; %{$self->{function_metadata}{$sch}{$pname}{$name}{metadata}} = %{$infos{$f}}; } $self->_restore_comments(\$self->{packages}{$pkg}{text}); } } #-------------------------------------------------------- my $default_global_vars = ''; my $number_fct = 0; my $i = 1; my $num_total_package = scalar keys %{$self->{packages}}; foreach my $pkg (sort keys %{$self->{packages}}) { my $total_size = 0; my $total_size_no_comment = 0; my $cost_value = 0; if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i, $num_total_package, 25, '=', 'packages', "generating $pkg" ), "\r"; } $i++, next if (!$self->{packages}{$pkg}{text} && !$self->{packages}{$pkg}{desc}); # Save and cleanup previous global variables defined in other package if (scalar keys %{$self->{global_variables}}) { foreach my $n (sort keys %{$self->{global_variables}}) { if (exists $self->{global_variables}{$n}{constant} || exists $self->{global_variables}{$n}{default}) { $default_global_vars .= "$n = '$self->{global_variables}{$n}{default}'\n"; } else { $default_global_vars .= "$n = ''\n"; } } } %{$self->{global_variables}} = (); my $pkgbody = ''; my $fct_cost = ''; if (!$self->{plsql_pgsql}) { $self->logit("Dumping package $pkg...\n", 1); if ($self->{file_per_function}) { my $f = "$dirprefix\L${pkg}\E_$self->{output}"; $f = "\L${pkg}\E_$self->{output}" if ($self->{psql_relative_path}); $f =~ s/\.(?:gz|bz2)$//i; $pkgbody = "\\i$self->{psql_relative_path} $f\n"; my $fhdl = $self->open_export_file("$dirprefix\L${pkg}\E_$self->{output}", 1); $self->set_binmode($fhdl) if (!$self->{compress}); $self->dump($sql_header . $self->{packages}{$pkg}{desc} . "\n\n" . $self->{packages}{$pkg}{text}, $fhdl); $self->close_export_file($fhdl); } else { $pkgbody = $self->{packages}{$pkg}{desc} . "\n\n" . $self->{packages}{$pkg}{text}; } } else { $self->{current_package} = $pkg; # If there is a declaration only do not go further than looking at global var and packages type if (!$self->{packages}{$pkg}{text}) { my $t = $self->_convert_package($pkg); $self->_restore_comments(\$t) if (!$self->{file_per_function}); $sql_output .= "\n\n-- Oracle package '$pkg' declaration, please edit to match PostgreSQL syntax.\n"; $sql_output .= $t . "\n"; $sql_output .= "-- End of Oracle package '$pkg' declaration\n\n"; if ($self->{estimate_cost}) { $sql_output .= "-- Total size of package code: $total_size bytes.\n"; $sql_output .= "-- Total size of package code without comments and header: $total_size_no_comment bytes.\n"; $sql_output .= "-- Detailed cost per function:\n" . $fct_cost; } $nothing++; $i++; next; } if ($self->{estimate_cost}) { $total_size += length($self->{packages}->{$pkg}{text}); } $self->_remove_comments(\$self->{packages}{$pkg}{text}); # Remove trailing comment and space $self->{packages}{$pkg}{text} =~ s/(\s*\%ORA2PG_COMMENT\d+\%)\s*$//s; $self->{packages}{$pkg}{text} =~ s/\s*$//s; # Normalyse package creation call $self->{packages}{$pkg}{text} =~ s/CREATE(?:\s+OR\s+REPLACE)?(?:\s+EDITIONABLE|\s+NONEDITIONABLE)?\s+PACKAGE\s+/CREATE OR REPLACE PACKAGE /is; if ($self->{estimate_cost}) { my %infos = $self->_lookup_package($self->{packages}{$pkg}{text}); foreach my $f (sort keys %infos) { next if (!$f); my @cnt = $infos{$f}{code} =~ /(\%ORA2PG_COMMENT\d+\%)/i; $total_size_no_comment += (length($infos{$f}{code}) - (17 * length(join('', @cnt)))); my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $infos{$f}{code}, $infos{$f}{type}); $self->logit("Function $f estimated cost: $cost\n", 1); $cost_value += $cost; $number_fct++; $fct_cost .= "\t-- Function $f total estimated cost: $cost\n"; foreach (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$_}); $fct_cost .= "\t\t-- $_ => $cost_detail{$_}"; if (!$self->{is_mysql}) { $fct_cost .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_SCORE{$_}); } else { $fct_cost .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_}); } $fct_cost .= "\n"; } } $cost_value += $Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}; $fct_cost .= "-- Total estimated cost for package $pkg: $cost_value units, " . $self->_get_human_cost($cost_value) . "\n"; } $txt = $self->_convert_package($pkg); $self->_restore_comments(\$txt) if (!$self->{file_per_function}); $txt =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; if (!$self->{file_per_function}) { $self->normalize_function_call(\$txt); } $pkgbody .= $txt; $pkgbody =~ s/[\r\n]*\bEND;\s*$//is; $pkgbody =~ s/(\s*;)\s*$/$1/is; } if ($self->{estimate_cost}) { $self->logit("Total size of package code: $total_size bytes.\n", 1); $self->logit("Total size of package code without comments and header: $total_size_no_comment bytes.\n", 1); $self->logit("Total estimated cost for package $pkg: $cost_value units, " . $self->_get_human_cost($cost_value) . ".\n", 1); } if ($pkgbody && ($pkgbody =~ /[a-z]/is)) { $sql_output .= "\n\n-- Oracle package '$pkg' declaration, please edit to match PostgreSQL syntax.\n"; $sql_output .= $pkgbody . "\n"; $sql_output .= "-- End of Oracle package '$pkg' declaration\n\n"; if ($self->{estimate_cost}) { $sql_output .= "-- Total size of package code: $total_size bytes.\n"; $sql_output .= "-- Total size of package code without comments and header: $total_size_no_comment bytes.\n"; $sql_output .= "-- Detailed cost per function:\n" . $fct_cost; } $nothing++; } $self->{total_pkgcost} += ($number_fct*$Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION'}); $self->{total_pkgcost} += $Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}; $i++; } if ($self->{estimate_cost} && $number_fct) { $self->logit("Total number of functions found inside all packages: $number_fct.\n", 1); } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_package, 25, '=', 'packages', 'end of output.'), "\n"; } if (!$nothing) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_output); if (scalar keys %{ $self->{object_dependencies} } > 0) { my $sp_tree = "object_type;object_name;routines_called;insert;update;delete;merge\n"; foreach my $caller ( sort keys %{ $self->{object_dependencies} } ) { $sp_tree .= "PACKAGE;$caller"; $sp_tree .= ";"; foreach my $sp (@{ $self->{object_dependencies}{$caller}{routines} }) { my $star = ($#{ $self->{object_dependencies}{$sp}{routines} } >= 0) ? '*' : ''; $sp_tree .= "$sp$star,"; } $sp_tree =~ s/,$//; $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{insert} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{update} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{delete} }); $sp_tree .= ";" . join(',', @{ $self->{object_dependencies}{$caller}{merge} }); $sp_tree .= "\n"; } my $fhdl = $self->open_export_file("packages_dependencies.csv"); $self->dump($sp_tree, $fhdl); $self->close_export_file($fhdl); } $self->{packages} = (); $sql_output = ''; # Create file to load custom variable initialization into postgresql.conf if (scalar keys %{$self->{global_variables}}) { foreach my $n (sort keys %{$self->{global_variables}}) { if (exists $self->{global_variables}{$n}{constant} || exists $self->{global_variables}{$n}{default}) { $default_global_vars .= "$n = '$self->{global_variables}{$n}{default}'\n"; } else { $default_global_vars .= "$n = ''\n"; } } } %{$self->{global_variables}} = (); # Save global variable that need to be initialized at startup if ($default_global_vars) { my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); open(OUT, ">${dirprefix}global_variables.conf"); print OUT "# Global variables with default values used in packages.\n"; print OUT $default_global_vars; close(OUT); } return; } =head2 export_type Export Oracle type into PostgreSQL compatible statement. =cut sub export_type { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add custom types definition...\n", 1); #--------------------------------------------------------- # Code to use to find type parser issues, it load a file # containing the untouched PL/SQL code from Oracle type #--------------------------------------------------------- if ($self->{input_file}) { $self->{types} = (); $self->logit("Reading input code from file $self->{input_file}...\n", 1); my $content = $self->read_input_file($self->{input_file}); $self->_remove_comments(\$content); my $i = 0; foreach my $l (split(/;/, $content)) { chomp($l); next if ($l =~ /^[\s\/]*$/s); my $cmt = ''; while ($l =~ s/(\%ORA2PG_COMMENT\d+\%)//s) { $cmt .= "$1"; } $self->_restore_comments(\$cmt); $l =~ s/^\s+//; $l =~ s/^CREATE\s+(?:OR REPLACE)?\s*(?:NONEDITIONABLE|EDITIONABLE)?\s*//is; $l .= ";\n"; if ($l =~ /^(SUBTYPE|TYPE)\s+([^\s\(]+)/is) { push(@{$self->{types}}, { ('name' => $2, 'code' => $l, 'comment' => $cmt, 'pos' => $i) }); } $i++; } } #-------------------------------------------------------- my $i = 1; foreach my $tpe (sort {$a->{pos} <=> $b->{pos} } @{$self->{types}}) { $self->logit("Dumping type $tpe->{name}...\n", 1); if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i, $#{$self->{types}}+1, 25, '=', 'types', "generating $tpe->{name}" ), "\r"; } if ($self->{plsql_pgsql}) { $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}); } else { $tpe->{code} =~ s/^CREATE TYPE/TYPE/i; if ($tpe->{code} !~ /^SUBTYPE\s+/) { $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; } } $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; $sql_output .= $tpe->{comment} . $tpe->{code} . "\n"; $i++; } $self->_restore_comments(\$sql_output); $self->{comment_values} = (); if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $#{$self->{types}}+1, 25, '=', 'types', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } if ($self->{export_schema} && ($self->{pg_schema} || $self->{schema})) { $sql_header .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . ";\n"; } $self->dump($sql_header . $sql_output); return; } =head2 export_tablespace Export Oracle tablespace into PostgreSQL compatible statement. =cut sub export_tablespace { my $self = shift; my $sql_header = $self->_set_file_header(); $sql_header .= "-- Oracle tablespaces export, please edit path to match your filesystem.\n"; $sql_header .= "-- In PostgreSQl the path must be a directory and is expected to already exists\n"; my $sql_output = ""; $self->logit("Add tablespaces definition...\n", 1); my $create_tb = ''; my @done = (); # Read DML from file if any if ($self->{input_file}) { $self->read_tablespace_from_file(); } my $dirprefix = ''; foreach my $tb_type (sort keys %{$self->{tablespaces}}) { #next if ($tb_type eq 'INDEX PARTITION' || $tb_type eq 'TABLE PARTITION'); # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME foreach my $tb_name (sort keys %{$self->{tablespaces}{$tb_type}}) { foreach my $tb_path (sort keys %{$self->{tablespaces}{$tb_type}{$tb_name}}) { # Replace Oracle tablespace filename my $loc = $tb_name; if ($tb_path =~ /^(.*[^\\\/]+)/) { $loc = $1 . '/' . $loc; } if (!grep(/^$tb_name$/, @done)) { $create_tb .= "CREATE TABLESPACE \L$tb_name\E LOCATION '$loc';\n"; my $owner = $self->{list_tablespaces}{$tb_name}{owner} || ''; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $create_tb .= "ALTER TABLESPACE " . $self->quote_object_name($tb_name) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } push(@done, $tb_name); foreach my $obj (@{$self->{tablespaces}{$tb_type}{$tb_name}{$tb_path}}) { $tb_type =~ s/ PARTITION//; next if ($self->{file_per_index} && ($tb_type eq 'INDEX')); $sql_output .= "ALTER $tb_type " . $self->quote_object_name($obj) . " SET TABLESPACE " . $self->quote_object_name($tb_name) . ";\n"; } } } } $sql_output = "$create_tb\n" . $sql_output if ($create_tb); if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); if ($self->{file_per_index} && (scalar keys %{$self->{tablespaces}} > 0)) { my $fhdl = undef; $self->logit("Dumping tablespace alter indexes to one separate file : TBSP_INDEXES_$self->{output}\n", 1); $fhdl = $self->open_export_file("TBSP_INDEXES_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $sql_output = ''; foreach my $tb_type (sort keys %{$self->{tablespaces}}) { # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME foreach my $tb_name (sort keys %{$self->{tablespaces}{$tb_type}}) { foreach my $tb_path (sort keys %{$self->{tablespaces}{$tb_type}{$tb_name}}) { # Replace Oracle tablespace filename my $loc = $tb_name; $tb_path =~ /^(.*)[^\\\/]+$/; $loc = $1 . $loc; foreach my $obj (@{$self->{tablespaces}{$tb_type}{$tb_name}{$tb_path}}) { $tb_type =~ s/ PARTITION//; next if ($tb_type eq 'TABLE'); $sql_output .= "ALTER $tb_type \L$obj\E SET TABLESPACE \L$tb_name\E;\n"; } } } } $sql_output = "-- Nothing found of type $self->{type}\n" if (!$sql_output && !$self->{no_header}); $self->dump($sql_header . $sql_output, $fhdl); $self->close_export_file($fhdl); } return; } =head2 export_kettle Export Oracle table into Kettle script to load data into PostgreSQL. =cut sub export_kettle { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; # Remove external table from data export if (scalar keys %{$self->{external_table}} ) { foreach my $table (keys %{$self->{tables}}) { if ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { delete $self->{tables}{$table}; } } } # Ordering tables by name by default my @ordered_tables = sort { $a cmp $b } keys %{$self->{tables}}; if (lc($self->{data_export_order}) eq 'size') { @ordered_tables = sort { ($self->{tables}{$b}{table_info}{num_rows} || $self->{tables}{$a}{table_info}{num_rows}) ? $self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows} : $a cmp $b } keys %{$self->{tables}}; } # User provide the ordered list of table from a file elsif (-e $self->{data_export_order}) { if (open(my $tfh, '<', $self->{data_export_order})) { @ordered_tables = (); while (my $l = <$tfh>) { chomp($l); next if (!exists $self->{tables}{$!}); push(@ordered_tables, $l); } close($tfh); } else { $self->logit("FATAL: can't read file $self->{data_export_order} for ordering table export. $!\n", 0, 1); } } my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); foreach my $table (@ordered_tables) { $shell_commands .= $self->create_kettle_output($table, $dirprefix); } $self->dump("#!/bin/sh\n\n", $fhdl); $self->dump("KETTLE_TEMPLATE_PATH='.'\n\n", $fhdl); $self->dump($shell_commands, $fhdl); return; } =head2 export_partition Export Oracle partition into PostgreSQL compatible statement. =cut sub export_partition { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add partitions definition...\n", 1); my $total_partition = 0; foreach my $t (sort keys %{ $self->{partitions} }) { $total_partition += scalar keys %{$self->{partitions}{$t}}; } foreach my $t (sort keys %{ $self->{subpartitions_list} }) { foreach my $p (sort keys %{ $self->{subpartitions_list}{$t} }) { $total_partition += $self->{subpartitions_list}{$t}{$p}{count}; } } # Extract partition definition from partitioned tables my $nparts = 1; my $partition_indexes = ''; foreach my $table (sort keys %{$self->{partitions}}) { my $function = ''; $function = qq{ CREATE$self->{create_or_replace} FUNCTION ${table}_insert_trigger() RETURNS TRIGGER AS \$\$ BEGIN } if (!$self->{pg_supports_partition}); my $cond = 'IF'; my $funct_cond = ''; my %create_table = (); my $idx = 0; my $old_pos = ''; my $old_part = ''; my $owner = ''; my $PGBAR_REFRESH = set_refresh_count($total_partition); # Extract partitions in their position order foreach my $pos (sort {$a <=> $b} keys %{$self->{partitions}{$table}}) { next if (!$self->{partitions}{$table}{$pos}{name}); my $part = $self->{partitions}{$table}{$pos}{name}; if (!$self->{quiet} && !$self->{debug} && ($nparts % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($nparts, $total_partition, 25, '=', 'partitions', "generating $table/$part" ), "\r"; } $nparts++; my $create_table_tmp = ''; my $create_table_index_tmp = ''; my $tb_name = ''; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $tb_name = $self->{replaced_tables}{lc($table)}; } if ($self->{rename_partition}) { $tb_name = ($tb_name||$table) . '_part' . $pos; } else { if ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { $tb_name = $1 . '.' . $part; } else { $tb_name = $part; } } $tb_name = $table . '_default' if (!$tb_name); $create_table_tmp .= "DROP TABLE $self->{pg_supports_ifexists} " . $self->quote_object_name($tb_name) . ";\n" if ($self->{drop_if_exists}); if (!$self->{pg_supports_partition}) { if (!exists $self->{subpartitions}{$table}{$part}) { $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " ( CHECK (\n"; } } else { $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " PARTITION OF " . $self->quote_object_name($table) . "\n"; $create_table_tmp .= "FOR VALUES"; } my @condition = (); my @ind_col = (); my $check_cond = ''; my $reftable = ''; for (my $i = 0; $i <= $#{$self->{partitions}{$table}{$pos}{info}}; $i++) { $reftable = $table; if ($self->{partition_by_reference} eq 'duplicate' && $self->{partitions}{$table}{$pos}{info}[$i]->{refrtable}) { $reftable = $self->{partitions}{$table}{$pos}{info}[$i]->{refrtable}; } # We received all values for partitonning on multiple column, so get the one at the right indice my $value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value}); if ($#{$self->{partitions}{$reftable}{$pos}{info}} >= 0) { my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value})); $value = $values[$i]; } my $old_value = ''; if ($old_part) { $old_value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$old_pos}{info}[$i]->{value}); if ($#{$self->{partitions}{$reftable}{$pos}{info}} == 0) { my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$old_pos}{info}[$i]->{value})); $old_value = $values[$i]; } } if ($self->{partitions}{$reftable}{$pos}{info}[$i]->{type} eq 'LIST') { $value = $self->{partitions}{$reftable}{$pos}{info}[$i]->{value}; if (!$self->{pg_supports_partition}) { $check_cond .= "\t$self->{partitions}{$reftable}{$pos}{info}[$i]->{column} IN ($value)"; } else { $check_cond .= " IN ($value)"; } } elsif ($self->{partitions}{$reftable}{$pos}{info}[$i]->{type} eq 'RANGE') { if (!$self->{pg_supports_partition}) { if ($old_part eq '') { $check_cond .= "\t$self->{partitions}{$reftable}{$pos}{info}[$i]->{column} < $value"; } else { $check_cond .= "\t$self->{partitions}{$reftable}{$pos}{info}[$i]->{column} >= $old_value" . " AND $self->{partitions}{$reftable}{$pos}{info}[$i]->{column} < $value"; } } else { if ($old_part eq '') { my $val = 'MINVALUE,' x ($#{$self->{partitions}{$reftable}{$pos}{info}}+1); $val =~ s/,$//; $check_cond .= " FROM ($val) TO ($value)"; } else { $check_cond .= " FROM ($old_value) TO ($value)"; } $i += $#{$self->{partitions}{$reftable}{$pos}{info}}; } } elsif ($self->{partitions}{$reftable}{$pos}{info}[$i]->{type} eq 'HASH') { if ($self->{pg_version} < 11) { print STDERR "WARNING: Hash partitioning not supported, skipping partitioning of table $reftable\n"; $function = ''; $create_table_tmp = ''; $create_table_index_tmp = ''; next; } else { my $part_clause = " WITH (MODULUS " . (scalar keys %{$self->{partitions}{$reftable}}) . ", REMAINDER " . ($pos-1) . ")"; $check_cond .= $part_clause if ($check_cond !~ /\Q$part_clause\E$/); } } elsif ($reftable eq $table) { print STDERR "WARNING: Unknown partitioning type $self->{partitions}{$reftable}{$pos}{info}[$i]->{type}, skipping partitioning of table $reftable\n"; $create_table_tmp = ''; $create_table_index_tmp = ''; next; } if (!$self->{pg_supports_partition}) { $check_cond .= " AND" if ($i < $#{$self->{partitions}{$reftable}{$pos}{info}}); } my $fct = ''; my $colname = $self->{partitions}{$reftable}{$pos}{info}[$i]->{column}; if ($colname =~ s/([^\(]+)\(([^\)]+)\)/$2/) { $fct = $1; } my $cindx = $self->{partitions}{$reftable}{$pos}{info}[$i]->{column} || ''; $cindx = lc($cindx) if (!$self->{preserve_case}); $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); my $has_hash_subpartition = 0; if (exists $self->{subpartitions}{$reftable}{$part}) { foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$reftable}{$part}}) { for (my $j = 0; $j <= $#{$self->{subpartitions}{$reftable}{$part}{$p}{info}}; $j++) { if ($self->{subpartitions}{$reftable}{$part}{$p}{info}[$j]->{type} eq 'HASH') { $has_hash_subpartition = 1; last; } } last if ($has_hash_subpartition); } } if (!exists $self->{subpartitions}{$reftable}{$part} || (!$self->{pg_supports_partition} && $has_hash_subpartition)) { # Reproduce indexes definition from the main table before PG 11 # after they are automatically created on partition tables if ($self->{pg_version} < 11) { my ($idx, $fts_idx) = $self->_create_indexes($reftable, 0, %{$self->{tables}{$reftable}{indexes}}); my $tb_name2 = $self->quote_object_name($tb_name); if ($cindx) { $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("${tb_name}_$colname$pos") . " ON " . $self->quote_object_name($tb_name) . " ($cindx);\n"; } if ($idx || $fts_idx) { $idx =~ s/ $reftable/ $tb_name2/igs; $fts_idx =~ s/ $reftable/ $tb_name2/igs; # remove indexes already created $idx =~ s/CREATE [^;]+ \($cindx\);//; $fts_idx =~ s/CREATE [^;]+ \($cindx\);//; if ($idx || $fts_idx) { # fix index name to avoid duplicate index name $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; $fts_idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; $create_table_index_tmp .= "-- Reproduce partition indexes that was defined on the parent table\n"; } $create_table_index_tmp .= "$idx\n" if ($idx); $create_table_index_tmp .= "$fts_idx\n" if ($fts_idx); } # Set the unique (and primary) key definition $idx = $self->_create_unique_keys($reftable, $self->{tables}{$reftable}{unique_key}); if ($idx) { $idx =~ s/ $reftable/ $tb_name2/igs; # remove indexes already created $idx =~ s/CREATE [^;]+ \($cindx\);//; if ($idx) { # fix index name to avoid duplicate index name $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; $create_table_index_tmp .= "-- Reproduce partition unique indexes / pk that was defined on the parent table\n"; $create_table_index_tmp .= "$idx\n"; # Remove duplicate index with this one if ($idx =~ /ALTER TABLE $tb_name2 ADD PRIMARY KEY (.*);/s) { my $collist = quotemeta($1); $create_table_index_tmp =~ s/CREATE INDEX [^;]+ ON $tb_name2 $collist;//s; } } } } } my $deftb = $self->{partitions_default}{$reftable}{name}; $deftb = $reftable . '_part_default' if ($self->{rename_partition}); # Reproduce indexes definition from the main table before PG 11 # after they are automatically created on partition tables if ($self->{pg_version} < 11) { if (exists $self->{partitions_default}{$reftable} && ($create_table{$reftable}{index} !~ /ON $deftb /)) { $cindx = $self->{partitions}{$reftable}{$pos}{info}[$i]->{column} || ''; $cindx = lc($cindx) if (!$self->{preserve_case}); $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); if ($cindx) { $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name($deftb . '_' . $colname) . " ON " . $self->quote_object_name($deftb) . " ($cindx);\n"; } } push(@ind_col, $self->{partitions}{$reftable}{$pos}{info}[$i]->{column}) if (!grep(/^$self->{partitions}{$reftable}{$pos}{info}[$i]->{column}$/, @ind_col)); } if ($self->{partitions}{$reftable}{$pos}{info}[$i]->{type} eq 'LIST') { if (!$fct) { push(@condition, "NEW.$self->{partitions}{$reftable}{$pos}{info}[$i]->{column} IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value}) . ")"); } else { push(@condition, "$fct(NEW.$colname) IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value}) . ")"); } } elsif ($self->{partitions}{$reftable}{$pos}{info}[$i]->{type} eq 'RANGE') { if (!$fct) { push(@condition, "NEW.$self->{partitions}{$reftable}{$pos}{info}[$i]->{column} < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value})); } else { push(@condition, "$fct(NEW.$colname) < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$reftable}{$pos}{info}[$i]->{value})); } } $owner = $self->{partitions}{$reftable}{$pos}{info}[$i]->{owner} || ''; } if (!$self->{pg_supports_partition}) { if ($self->{partitions}{$table}{$pos}{info}[$i]->{type} ne 'HASH') { if (!exists $self->{subpartitions}{$table}{$part}) { $create_table_tmp .= $check_cond . "\n"; $create_table_tmp .= ") ) INHERITS ($table);\n"; } $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $create_table_tmp .= "ALTER TABLE " . $self->quote_object_name($tb_name) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } } else { $create_table_tmp .= $check_cond; if (exists $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type}) { $create_table_tmp .= "\nPARTITION BY " . $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type} . " ("; my $expr = ''; if (exists $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{columns}) { my $len = $#{$self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{columns}}; for (my $j = 0; $j <= $len; $j++) { if ($self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type} eq 'LIST') { $expr .= ' || ' if ($j > 0); } else { $expr .= ', ' if ($j > 0); } $expr .= $self->quote_object_name($self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{columns}[$j]); if ($self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type} eq 'LIST' && $len > 0) { $expr .= '::text'; } } if ($self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type} eq 'LIST' && $len >= 0) { $expr = '(' . $expr . ')'; } } else { if ($self->{plsql_pgsql}) { $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{expression} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{expression}); } $expr .= $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{expression}; } $expr = '(' . $expr . ')' if ($expr =~ /[\(\+\-\*\%:]/ && $expr !~ /^\(.*\)$/); $create_table_tmp .= "$expr)"; } $create_table_tmp .= ";\n" if ($create_table_tmp); } # Add subpartition if any defined on Oracle my $sub_funct_cond = ''; my $sub_old_part = ''; if (exists $self->{subpartitions}{$table}{$part}) { my $sub_cond = 'IF'; my $sub_funct_cond_tmp = ''; my $create_subtable_tmp = ''; my $total_subpartition = scalar %{$self->{subpartitions}{$table}{$part}} || 0; foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part}}) { my $subpart = $self->{subpartitions}{$table}{$part}{$p}{name}; my $sub_tb_name = $subpart; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any if ($self->{rename_partition}) { $sub_tb_name = $tb_name . '_subpart' . $p; } else { $sub_tb_name = "${table}_$sub_tb_name"; } if (!$self->{quiet} && !$self->{debug} && ($nparts % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($nparts, $total_partition, 25, '=', 'partitions', "generating $table/$part/$subpart" ), "\r"; } $nparts++; $create_subtable_tmp .= "DROP TABLE $self->{pg_supports_ifexists} " . $self->quote_object_name($sub_tb_name) . ";\n" if ($self->{drop_if_exists}); $create_subtable_tmp .= "CREATE TABLE " . $self->quote_object_name($sub_tb_name); if (!$self->{pg_supports_partition}) { $create_subtable_tmp .= " ( CHECK (\n"; } else { $create_subtable_tmp .= " PARTITION OF " . $self->quote_object_name($tb_name) . " -- $reftable -> $table\n"; $create_subtable_tmp .= "FOR VALUES"; } my $sub_check_cond_tmp = ''; my @subcondition = (); for (my $i = 0; $i <= $#{$self->{subpartitions}{$table}{$part}{$p}{info}}; $i++) { # We received all values for partitonning on multiple column, so get the one at the right indice my $value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}); if ($#{$self->{subpartitions}{$table}{$part}{$p}{info}} == 0) { my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); $value = $values[$i]; } my $old_value = ''; if ($sub_old_part) { $old_value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{value}); if ($#{$self->{subpartitions}{$table}{$part}{$p}{info}} == 0) { my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{value})); $old_value = $values[$i]; } } if ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'LIST') { if (!$self->{pg_supports_partition}) { $sub_check_cond_tmp .= "$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} IN ($value)"; } else { $sub_check_cond_tmp .= " IN ($value)"; } } elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'RANGE') { if (!$self->{pg_supports_partition}) { if ($old_part eq '') { $sub_check_cond_tmp .= "\t$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < $value"; } else { $sub_check_cond_tmp .= "\t$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} >= $old_value" . " AND $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < $value"; } } else { if ($old_part eq '') { my $val = 'MINVALUE,' x ($#{$self->{subpartitions}{$table}{$part}{$p}{info}}+1); $val =~ s/,$//; $sub_check_cond_tmp .= " FROM ($val) TO ($value)"; } else { $sub_check_cond_tmp .= " FROM ($old_value) TO ($value)"; } $i += $#{$self->{subpartitions}{$table}{$part}{$p}{info}}; } } elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'HASH') { if ($self->{pg_version} < 11) { print STDERR "WARNING: Hash partitioning not supported, skipping subpartitioning of table $table\n"; $create_subtable_tmp = ''; $sub_funct_cond_tmp = ''; next; } else { my $part_clause = " WITH (MODULUS " . (scalar keys %{$self->{subpartitions}{$table}{$part}}) . ", REMAINDER " . ($p-1) . ")"; $sub_check_cond_tmp .= $part_clause if ($sub_check_cond_tmp !~ /\Q$part_clause\E$/); } } else { print STDERR "WARNING: Unknown partitioning type $self->{partitions}{$table}{$pos}{info}[$i]->{type}, skipping partitioning of table $table\n"; $create_subtable_tmp = ''; $sub_funct_cond_tmp = ''; next; } if (!$self->{pg_supports_partition}) { $sub_check_cond_tmp .= " AND " if ($i < $#{$self->{subpartitions}{$table}{$part}{$p}{info}}); } # Reproduce indexes definition from the main table before PG 11 # after they are automatically created on partition tables if ($self->{pg_version} < 11) { push(@ind_col, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}) if (!grep(/^$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}$/, @ind_col)); my $fct = ''; my $colname = $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}; if ($colname =~ s/([^\(]+)\(([^\)]+)\)/$2/) { $fct = $1; } $cindx = join(',', @ind_col); $cindx = lc($cindx) if (!$self->{preserve_case}); $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); if ($cindx) { $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("${sub_tb_name}_$colname$p") . " ON " . $self->quote_object_name("$sub_tb_name") . " ($cindx);\n"; } my $tb_name2 = $self->quote_object_name("$sub_tb_name"); # Reproduce indexes definition from the main table my ($idx, $fts_idx) = $self->_create_indexes($table, 0, %{$self->{tables}{$table}{indexes}}); if ($idx || $fts_idx) { $idx =~ s/ $table/ $tb_name2/igs; $fts_idx =~ s/ $table/ $tb_name2/igs; # remove indexes already created $idx =~ s/CREATE [^;]+ \($cindx\);//; $fts_idx =~ s/CREATE [^;]+ \($cindx\);//; if ($idx || $fts_idx) { # fix index name to avoid duplicate index name $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; $fts_idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; $create_table_index_tmp .= "-- Reproduce subpartition indexes that was defined on the parent table\n"; } $create_table_index_tmp .= "$idx\n" if ($idx); $create_table_index_tmp .= "$fts_idx\n" if ($fts_idx); } # Set the unique (and primary) key definition $idx = $self->_create_unique_keys($table, $self->{tables}{$table}{unique_key}, $part); if ($idx) { $create_table_index_tmp .= "-- Reproduce subpartition unique indexes / pk that was defined on the parent table\n"; $idx =~ s/ $table/ $tb_name2/igs; # remove indexes already created $idx =~ s/CREATE [^;]+ \($cindx\);//; if ($idx) { # fix index name to avoid duplicate index name $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; $create_table_index_tmp .= "$idx\n"; # Remove duplicate index with this one if ($idx =~ /ALTER TABLE $tb_name2 ADD PRIMARY KEY (.*);/s) { my $collist = quotemeta($1); $create_table_index_tmp =~ s/CREATE INDEX [^;]+ ON $tb_name2 $collist;//s; } } } } if ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'LIST') { if (!$fct) { push(@subcondition, "NEW.$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}) . ")"); } else { push(@subcondition, "$fct(NEW.$colname) IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}) . ")"); } } elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'RANGE') { if (!$fct) { push(@subcondition, "NEW.$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); } else { push(@subcondition, "$fct(NEW.$colname) < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); } } $owner = $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{owner} || ''; } if ($self->{pg_supports_partition}) { $sub_check_cond_tmp .= ';'; $create_subtable_tmp .= "$sub_check_cond_tmp\n"; } else { $create_subtable_tmp .= $check_cond; $create_subtable_tmp .= " AND $sub_check_cond_tmp" if ($sub_check_cond_tmp); $create_subtable_tmp .= "\n) ) INHERITS ($table);\n"; } $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $create_subtable_tmp .= "ALTER TABLE " . $self->quote_object_name("$sub_tb_name") . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } if ($#subcondition >= 0) { $sub_funct_cond_tmp .= "\t\t$sub_cond ( " . join(' AND ', @subcondition) . " ) THEN INSERT INTO " . $self->quote_object_name("$sub_tb_name") . " VALUES (NEW.*);\n"; $sub_cond = 'ELSIF'; } $sub_old_part = $part; $sub_old_pos = $p; } if ($create_subtable_tmp) { $create_table_tmp .= $create_subtable_tmp; $sub_funct_cond = $sub_funct_cond_tmp; } } $check_cond = ''; # Fix case where default partition is taken as a value $create_table_tmp =~ s/FOR VALUES IN \(default\)/DEFAULT/igs; if ($#condition >= 0) { if (!$sub_funct_cond) { $funct_cond .= "\t$cond ( " . join(' AND ', @condition) . " ) THEN INSERT INTO " . $self->quote_object_name($tb_name) . " VALUES (NEW.*);\n"; } else { my $sub_old_pos = 0; if (!$self->{pg_supports_partition}) { $sub_funct_cond = Ora2Pg::PLSQL::convert_plsql_code($self, $sub_funct_cond); $funct_cond .= "\t$cond ( " . join(' AND ', @condition) . " ) THEN \n"; $funct_cond .= $sub_funct_cond; if (exists $self->{subpartitions_default}{$table}{$part}{name}) { my $deftb = $self->{subpartitions_default}{$table}{$part}{name}; $deftb = $table . '_part'. $pos . '_subpart_default' if ($self->{rename_partition}); $funct_cond .= "\t\tELSE INSERT INTO " . $self->quote_object_name($deftb) . " VALUES (NEW.*);\n\t\tEND IF;\n"; $create_table_tmp .= "DROP TABLE $self->{pg_supports_ifexists} " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}{name}") . ";\n" if ($self->{drop_if_exists}); $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}{name}") . " () INHERITS ($table);\n"; if ($cindx) { $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}{name}_$pos") . " ON " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}{name}") . " ($cindx);\n"; } } else { $funct_cond .= qq{ ELSE -- Raise an exception RAISE EXCEPTION 'Value out of range in subpartition. Fix the ${table}_insert_trigger() function!'; }; $funct_cond .= "\t\tEND IF;\n"; } } # With default partition just add default and continue elsif (exists $self->{subpartitions_default}{$table}{$part}) { my $tb_name = $self->{subpartitions_default}{$table}{$part}{name}; if ($self->{rename_partition}) { $tb_name = $table . '_part' . $pos . '_subpart_default'; } elsif ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { $tb_name = $1 . '.' . $self->{subpartitions_default}{$table}{$part}{name}; } $tb_name = $table . '_part' . $pos . '_subpart_default' if (!$tb_name); $create_table_tmp .= "DROP TABLE $self->{pg_supports_ifexists} " . $self->quote_object_name($tb_name) . ";\n" if ($self->{drop_if_exists}); if ($self->{pg_version} >= 11) { $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " PARTITION OF " . $self->quote_object_name($table) . " DEFAULT;\n"; } elsif ($self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{type} eq 'RANGE') { $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " PARTITION OF " . $self->quote_object_name($table) . " FOR VALUES FROM ($self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[-1]->{value}) TO (MAX_VALUE);\n"; } } } $cond = 'ELSIF'; } $old_part = $part; $old_pos = $pos; $create_table{$table}{table} .= $create_table_tmp; $create_table{$table}{index} .= $create_table_index_tmp; } if (exists $create_table{$table}) { if (!$self->{pg_supports_partition}) { if (exists $self->{partitions_default}{$table} && scalar keys %{$self->{partitions_default}{$table}} > 0) { my $deftb = $self->{partitions_default}{$table}{name}; $deftb = $table . '_part_default' if ($self->{rename_partition}); my $pname = $self->quote_object_name($deftb); $function .= $funct_cond . qq{ ELSE INSERT INTO $pname VALUES (NEW.*); }; } elsif ($function) { $function .= $funct_cond . qq{ ELSE -- Raise an exception RAISE EXCEPTION 'Value out of range in partition. Fix the ${table}_insert_trigger() function!'; }; } $function .= qq{ END IF; RETURN NULL; END; \$\$ LANGUAGE plpgsql; } if ($function); $function = Ora2Pg::PLSQL::convert_plsql_code($self, $function); } else { # With default partition just add default and continue if (exists $self->{partitions_default}{$table} && scalar keys %{$self->{partitions_default}{$table}} > 0) { my $tb_name = ''; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $tb_name = $self->{replaced_tables}{lc($table)}; } if ($self->{rename_partition}) { $tb_name = ($tb_name||$table) . '_part_default'; } else { if ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { $tb_name = $1 . '.' . $self->{partitions_default}{$table}{name}; } else { $tb_name = $self->{partitions_default}{$table}{name}; } } $tb_name = $table . '_part_default' if (!$tb_name); $create_table{$table}{table} .= "DROP TABLE $self->{pg_supports_ifexists} " . $self->quote_object_name($tb_name) . ";\n" if ($self->{drop_if_exists}); if ($self->{pg_version} >= 11) { $create_table{$table}{table} .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " PARTITION OF " . $self->quote_object_name($table) . " DEFAULT;\n"; } else { $create_table{$table}{table} .= "CREATE TABLE " . $self->quote_object_name($tb_name) . " PARTITION OF " . $self->quote_object_name($table) . " FOR VALUES FROM ($self->{partitions}{$table}{$old_pos}{info}[-1]->{value}) TO (MAX_VALUE);\n"; } } } } if (exists $create_table{$table}) { $partition_indexes .= qq{ -- Create indexes on each partition of table $table $create_table{$table}{index} } if ($create_table{$table}{index}); $sql_output .= qq{ $create_table{$table}{table} }; my $tb = $self->quote_object_name($table); my $trg = $self->quote_object_name("${table}_insert_trigger"); my $defname = $self->{partitions_default}{$table}{name}; $defname = $table . '_part_default' if ($self->{rename_partition}); $defname = $self->quote_object_name($defname); if (!$self->{pg_supports_partition} && $function) { $sql_output .= qq{ -- Create default table, where datas are inserted if no condition match CREATE TABLE $defname () INHERITS ($tb); } if ($self->{partitions_default}{$table}{name}); $sql_output .= qq{ $function CREATE TRIGGER ${table}_trigger_insert BEFORE INSERT ON $table FOR EACH ROW EXECUTE PROCEDURE $trg(); ------------------------------------------------------------------------------- }; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $sql_output .= "ALTER TABLE " . $self->quote_object_name($self->{partitions_default}{$table}{name}) . " OWNER TO " . $self->quote_object_name($owner) . ";\n" if ($self->{partitions_default}{$table}{name}); $sql_output .= "ALTER FUNCTION " . $self->quote_object_name("${table}_insert_trigger") . "() OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } } } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($nparts - 1, $total_partition, 25, '=', 'partitions', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $sql_output =~ s/\n{3,}/\n\n/g; $self->dump($sql_header . $sql_output); if ($partition_indexes) { my $fhdl = undef; $self->logit("Dumping partition indexes to file : PARTITION_INDEXES_$self->{output}\n", 1); $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; $sql_header .= "-- Copyright 2000-2025 Gilles DAROLD. All rights reserved.\n"; $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; $sql_header = '' if ($self->{no_header}); $fhdl = $self->open_export_file("PARTITION_INDEXES_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->dump($sql_header . $partition_indexes, $fhdl); $self->close_export_file($fhdl); } return; } =head2 export_synonym Export Oracle synonym into PostgreSQL compatible statement. =cut sub export_synonym { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Add synonyms definition...\n", 1); # Read DML from file if any if ($self->{input_file}) { $self->read_synonym_from_file(); } my $i = 1; my $num_total_synonym = scalar keys %{$self->{synonyms}}; my $count_syn = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_synonym); foreach my $syn (sort { $a cmp $b } keys %{$self->{synonyms}}) { if (!$self->{quiet} && !$self->{debug} && ($count_syn % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($i, $num_total_synonym, 25, '=', 'synonyms', "generating $syn" ), "\r"; } $count_syn++; if ($self->{synonyms}{$syn}{dblink}) { $sql_output .= "-- You need to create foreign table $self->{synonyms}{$syn}{table_owner}.$self->{synonyms}{$syn}{table_name} using foreign server: '$self->{synonyms}{$syn}{dblink}'\n -- see DBLINK export type to export the server definition\n"; } $sql_output .= "CREATE$self->{create_or_replace} VIEW " . $self->quote_object_name($syn) . " AS SELECT * FROM "; if ($self->{synonyms}{$syn}{table_owner} && !$self->{schema} && $self->{export_schema}) { $sql_output .= $self->quote_object_name("$self->{synonyms}{$syn}{table_owner}.$self->{synonyms}{$syn}{table_name}") . ";\n"; } else { $sql_output .= $self->quote_object_name($self->{synonyms}{$syn}{table_name}) . ";\n"; } if ($self->{force_owner}) { my $owner = $self->{synonyms}{$syn}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} && ($self->{force_owner} ne "1")); $sql_output .= "ALTER VIEW " . $self->quote_object_name("$self->{synonyms}{$syn}{owner}.$syn") . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } $i++; } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($i - 1, $num_total_synonym, 25, '=', 'synonyms', 'end of output.'), "\n"; } if (!$sql_output) { $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); } $self->dump($sql_header . $sql_output); return; } =head2 export_table Export Oracle table into PostgreSQL compatible statement. =cut sub export_table { my $self = shift; my $sql_header = $self->_set_file_header(); my $sql_output = ""; $self->logit("Exporting tables...\n", 1); if (!$self->{oracle_fdw_data_export}) { if ($self->{export_schema} && ($self->{schema} || $self->{pg_schema})) { if ($self->{create_schema}) { if ($self->{pg_schema} && $self->{pg_schema} =~ /,/) { $self->logit("FATAL: with export type TABLE you can not set multiple schema to PG_SCHEMA when EXPORT_SCHEMA is enabled.\n", 0, 1); } $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . ";\n"; } my $owner = ''; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $owner ||= $self->{schema}; if ($owner && $self->{create_schema}) { $sql_output .= "ALTER SCHEMA " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } $sql_output .= "\n"; } elsif ($self->{export_schema}) { if ($self->{create_schema}) { my $current_schema = ''; foreach my $table (sort keys %{$self->{tables}}) { if ($table =~ /^([^\.]+)\..*/) { if ($1 ne $current_schema) { $current_schema = $1; $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($1) . ";\n"; } } } } } $sql_output .= $self->set_search_path(); } # Read DML from file if any if ($self->{input_file}) { $self->read_schema_from_file(); } my $constraints = ''; if ($self->{file_per_constraint}) { $constraints .= $self->set_search_path(); } my $indices = ''; my $fts_indices = ''; # Find first the total number of tables my $num_total_table = scalar keys %{$self->{tables}}; # Hash that will contains virtual column information to build triggers my %virtual_trigger_info = (); # Stores DDL to restart autoincrement sequences my $sequence_output = ''; if ($self->{is_mssql}) { if (lc($self->{case_insensitive_search}) eq 'citext') { $sql_output .= "CREATE EXTENSION IF NOT EXISTS citext;\n"; } } else { $self->{case_insensitive_search} = 'none'; } # Dump all table/index/constraints SQL definitions my $ib = 1; my $count_table = 0; my $PGBAR_REFRESH = set_refresh_count($num_total_table); foreach my $table (sort { if (exists $self->{tables}{$a}{internal_id}) { $self->{tables}{$a}{internal_id} <=> $self->{tables}{$b}{internal_id}; } else { $a cmp $b; } } keys %{$self->{tables}}) { # Foreign table can not be temporary next if (($self->{type} eq 'FDW' || $self->{oracle_fdw_data_export}) and $self->{tables}{$table}{table_info}{type} =~/ TEMPORARY/); $self->logit("Dumping table $table...\n", 1); if (!$self->{quiet} && !$self->{debug} && ($count_table % $PGBAR_REFRESH) == 0) { print STDERR $self->progress_bar($ib, $num_total_table, 25, '=', 'tables', "exporting $table" ), "\r"; } $count_table++; # Create FDW server if required if ( $self->{external_to_fdw} && grep(/^$table$/i, keys %{$self->{external_table}}) ) { my $srv_name = "\L$self->{external_table}{$table}{directory}\E"; $srv_name =~ s/^.*\.//; $sql_header .= "CREATE EXTENSION IF NOT EXISTS file_fdw;\n\n" if ($sql_header !~ /CREATE EXTENSION .* file_fdw;/is); $sql_header .= "CREATE SERVER $srv_name FOREIGN DATA WRAPPER file_fdw;\n\n" if ($sql_header !~ /CREATE SERVER $srv_name FOREIGN DATA WRAPPER file_fdw;/is); } # MySQL ON UPDATE clause my %trigger_update = (); my $tbname = $self->get_replaced_tbname($table); my $foreign = ''; if ( ($self->{type} eq 'FDW') || $self->{oracle_fdw_data_export} || ($self->{external_to_fdw} && (grep(/^$table$/i, keys %{$self->{external_table}}) || $self->{tables}{$table}{table_info}{connection})) ) { $foreign = ' FOREIGN '; } my $obj_type = $self->{tables}{$table}{table_info}{type} || 'TABLE'; if ( ($obj_type eq 'TABLE') && $self->{tables}{$table}{table_info}{nologging} && !$self->{disable_unlogged} ) { $obj_type = 'UNLOGGED ' . $obj_type; } if ($self->{export_gtt} && !$foreign && $self->{tables}{$table}{table_info}{temporary} eq 'Y') { if ($sql_output !~ /LOAD '.*pgtt';/s) { if (!$self->{pgtt_nosuperuser}) { $sql_output .= "\nLOAD 'pgtt';\n"; } else { $sql_output .= "\nLOAD '\$libdir/plugins/pgtt';\n"; } } $obj_type = ' /*GLOBAL*/ TEMPORARY TABLE' if ($obj_type =~ /TABLE/); } if (exists $self->{tables}{$table}{table_as}) { if ($self->{plsql_pgsql}) { $self->{tables}{$table}{table_as} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{tables}{$table}{table_as}); } $sql_output .= "\nDROP${foreign} TABLE $self->{pg_supports_ifexists} $tbname;" if ($self->{drop_if_exists}); my $withoid = _make_WITH($self->{with_oid}, $self->{tables}{$tbname}{table_info}); $sql_output .= "\nCREATE $obj_type $tbname $withoid AS $self->{tables}{$table}{table_as};\n"; next; } if (exists $self->{tables}{$table}{truncate_table}) { $sql_output .= "\nTRUNCATE TABLE $tbname;\n"; } my $serial_sequence = ''; my $enum_str = ''; my @skip_column_check = (); $sql_output .= "#ORA2PGENUM#\n"; # used to insert any enum data type before the table that will use it if (exists $self->{tables}{$table}{column_info}) { my $schem = ''; # Add the destination schema if ($self->{oracle_fdw_data_export} && ($self->{type} eq 'INSERT' || $self->{type} eq 'COPY')) { $sql_output .= "\nCREATE FOREIGN TABLE $self->{fdw_import_schema}.$tbname (\n"; } else { $sql_output .= "\nDROP${foreign} TABLE $self->{pg_supports_ifexists} $tbname;" if ($self->{drop_if_exists}); $sql_output .= "\nCREATE$foreign $obj_type $tbname (\n"; } # get column name list. my @collist = (); foreach my $k (sort {$self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]} keys %{$self->{tables}{$table}{column_info}}) { push(@collist, $self->{tables}{$table}{column_info}{$k}[0]); } # Extract column information following the position order foreach my $k (sort { if (!$self->{reordering_columns}) { $self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]; } else { my $tmpa = $self->{tables}{$table}{column_info}{$a}; $tmpa->[2] =~ s/\D//g; my $typa = $self->_sql_type($tmpa->[1], $tmpa->[2], $tmpa->[5], $tmpa->[6], $tmpa->[4]); $typa =~ s/\(.*//; my $tmpb = $self->{tables}{$table}{column_info}{$b}; $tmpb->[2] =~ s/\D//g; my $typb = $self->_sql_type($tmpb->[1], $tmpb->[2], $tmpb->[5], $tmpb->[6], $tmpb->[4]); $typb =~ s/\(.*//; if($TYPALIGN{$typa} != $TYPALIGN{$typb}){ # sort by field size asc $TYPALIGN{$typa} <=> $TYPALIGN{$typb}; } else { # if same size sort by original position $self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]; } } } keys %{$self->{tables}{$table}{column_info}}) { # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE my $f = $self->{tables}{$table}{column_info}{$k}; $f->[2] =~ s/[^0-9\-\.]//g; my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4], 1); $type = "$f->[1], $f->[2]" if (!$type); # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"}; } # Check if we need auto increment if ($f->[12] eq 'auto_increment' || $f->[12] eq '1') { if ($type !~ s/bigint/bigserial/) { if ($type !~ s/smallint/smallserial/) { $type =~ s/integer/serial/; $type =~ s/numeric.*/bigserial/; } } if ($type =~ /serial/) { my $seqname = lc($tbname) . '_' . lc($fname) . '_seq'; if ($self->{preserve_case}) { $seqname = $tbname . '_' . $fname . '_seq'; } my $tobequoted = 0; if ($seqname =~ s/"//g) { $tobequoted = 1; } if (length($seqname) > 63) { if (length($tbname) > 29) { $seqname = substr(lc($tbname), 0, 29); } else { $seqname = lc($tbname); } if (length($fname) > 29) { $seqname .= '_' . substr(lc($fname), 0, 29); } else { $seqname .= '_' . lc($fname); } $seqname .= '_seq'; } if ($tobequoted) { $seqname = '"' . $seqname . '"'; } if (exists $self->{tables}{$table}{table_info}{auto_increment}) { $self->{tables}{$table}{table_info}{auto_increment} = 1 if ($self->{is_mysql} && !$self->{tables}{$table}{table_info}{auto_increment}); $serial_sequence .= "ALTER SEQUENCE $seqname RESTART WITH $self->{tables}{$table}{table_info}{auto_increment};\n"; } } } # Check if this column should be replaced by a boolean following table/column name my $was_enum = 0; if ($f->[1] =~ s/^\s*ENUM\s*\(//i) { $was_enum = 1; $f->[1] =~ s/\)$//; my $keyname = lc($tbname . '_' . $fname . '_t'); $keyname =~ s/["\`]//g; $enum_str .= "\nCREATE TYPE " . $self->quote_object_name($keyname) . " AS ENUM ($f->[1]);"; $type = $self->quote_object_name($keyname); } my $typlen = $f->[5]; $typlen ||= $f->[2]; if (!$self->{oracle_fdw_data_export}) { if (grep(/^$f->[0]$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { $type = 'boolean'; push(@skip_column_check, $fname); } # Check if this column should be replaced by a boolean following type/precision elsif (exists $self->{'replace_as_boolean'}{uc($f->[1])} && ($self->{'replace_as_boolean'}{uc($f->[1])}[0] == $typlen)) { $type = 'boolean'; push(@skip_column_check, $fname); } } if ($f->[1] =~ /SDO_GEOMETRY/) { # 12:SRID,13:SDO_DIM,14:SDO_GTYPE # Set the dimension, array is (srid, dims, gtype) my $suffix = ''; if ($f->[13] == 3) { $suffix = 'Z'; } elsif ($f->[13] == 4) { $suffix = 'ZM'; } my $gtypes = ''; if (!$f->[14] || ($f->[14] =~ /,/) ) { $gtypes = $Ora2Pg::Oracle::ORA2PG_SDO_GTYPE{0}; } else { $gtypes = $f->[14]; } $type = "geometry($gtypes$suffix"; if ($f->[12]) { $type .= ",$f->[12]"; } $type .= ")"; } $type = $self->{'modify_type'}{"\L$table\E"}{"\L$f->[0]\E"} if (exists $self->{'modify_type'}{"\L$table\E"}{"\L$f->[0]\E"}); $fname = $self->quote_object_name($fname); my $citext_constraint = ''; if ($self->{case_insensitive_search} =~ /^citext$/i && $type =~ /^(?:char|varchar|text)/) { if ($type =~ /^(?:char|varchar|text)\s*\((\d+)\)/) { $constraints .= "ALTER TABLE " . $self->quote_object_name($table) . " ADD CHECK (char_length($fname) <= $1);\n"; } $type = 'citext'; } elsif ($self->{case_insensitive_search} !~ /^none$/i && $type =~ /^(?:char|varchar|text)/) { $type .= ' COLLATE ' . $self->{case_insensitive_search}; } $sql_output .= "\t$fname $type"; if ($foreign && $self->is_primary_key_column($table, $f->[0])) { $sql_output .= " OPTIONS (key 'true')"; } if (!$f->[3] || ($f->[3] =~ /^N/)) { # smallserial, serial and bigserial use a NOT NULL sequence as default value, # so we don't need to add it here if ($type !~ /serial/) { push(@{$self->{tables}{$table}{check_constraint}{notnull}}, $f->[0]); $sql_output .= " NOT NULL"; } } # Autoincremented columns if (!$self->{schema} && $self->{export_schema} && $f->[8] !~ /\./) { $f->[8] = "$f->[9].$f->[8]"; } if (exists $self->{identity_info}{$f->[8]}{$f->[0]} and $self->{type} ne 'FDW' and !$self->{oracle_fdw_data_export}) { $sql_output =~ s/ NOT NULL\s*$//s; # IDENTITY or serial column are NOT NULL by default if ($self->{pg_supports_identity}) { $sql_output =~ s/ [^\s]+$/ bigint/ if ($self->{force_identity_bigint}) ; # Force bigint $sql_output .= " GENERATED $self->{identity_info}{$f->[8]}{$f->[0]}{generation} AS IDENTITY"; if (exists $self->{identity_info}{$f->[8]}{$f->[0]}{options} && $self->{identity_info}{$f->[8]}{$f->[0]}{options} ne '') { # Adapt automatically the max value following the data type if ($sql_output =~ / (integer|int4|int) GENERATED/i) { $self->{identity_info}{$f->[8]}{$f->[0]}{options} =~ s/ 9223372036854775807/ 2147483647/s; } $sql_output .= " (" . $self->{identity_info}{$f->[8]}{$f->[0]}{options} . ')'; } } else { $sql_output =~ s/bigint\s*$/bigserial/s; $sql_output =~ s/smallint\s*$/smallserial/s; $sql_output =~ s/(integer|int)\s*$/serial/s; } $sql_output .= ",\n"; if ($self->{preserve_case}) { $sequence_output .= "SELECT ora2pg_upd_autoincrement_seq('$f->[8]','$f->[0]');\n"; } else { $sequence_output .= "SELECT ora2pg_upd_autoincrement_seq('\L$f->[8]\E','\L$f->[0]\E');\n"; } next; } # Default value if ($f->[4] ne "" && uc($f->[4]) ne 'NULL') { $f->[4] =~ s/^\s+//; $f->[4] =~ s/\s+$//; $f->[4] =~ s/"//gs; if ($self->{plsql_pgsql}) { $f->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $f->[4]); } # Check if the column make reference to an other column #my $use_other_col = 0; #foreach my $c (@collist) { # $use_other_col = 1 if ($f->[4] =~ /\b$c\b/i); #} if ($f->[10] eq 'YES' && $self->{pg_version} >= 12) { $sql_output .= " GENERATED ALWAYS AS (" . $f->[4] . ") STORED"; } # Check if this is a virtual column before proceeding to default value export elsif ($self->{tables}{$table}{column_info}{$k}[10] eq 'YES') { $virtual_trigger_info{$table}{$k} = $f->[4]; $virtual_trigger_info{$table}{$k} =~ s/"//gs; foreach my $c (keys %{$self->{tables}{$table}{column_info}}) { $virtual_trigger_info{$table}{$k} =~ s/\b$c\b/NEW.$c/gs; } } else { if (($f->[4] ne '') && ($self->{type} ne 'FDW') && !$self->{oracle_fdw_data_export}) { if ($type eq 'boolean') { my $found = 0; foreach my $k (sort {$b cmp $a} keys %{ $self->{ora_boolean_values} }) { if ($f->[4] =~ /\b$k\b/i) { $sql_output .= " DEFAULT '" . $self->{ora_boolean_values}{$k} . "'"; $found = 1; last; } } $sql_output .= " DEFAULT " . $f->[4] if (!$found); } else { if (($f->[4] !~ /^'/) && ($f->[4] =~ /[^\d\.]/)) { if ($type =~ /CHAR|TEXT/i || ($was_enum && $f->[1] =~ /'/i)) { $f->[4] = "'$f->[4]'" if ($f->[4] !~ /[']/ && $f->[4] !~ /\(.*\)/ && uc($f->[4]) ne 'NULL'); } elsif ($type =~ /DATE|TIME/i) { # Case of MSSQL datetime default value to 0, must be converted to '1900-01-01 00:00:00' if ($self->{is_mssql} && $f->[4] eq "(0)") { if ($type =~ /TIME/i) { $f->[4] = '1900-01-01 00:00:00'; } else { $f->[4] = '1900-01-01'; } } # All other cases if ($f->[4] =~ /^0000-/) { if ($self->{replace_zero_date}) { $f->[4] = $self->{replace_zero_date}; } else { $f->[4] =~ s/^0000-\d+-\d+/1970-01-01/; } } if ($f->[4] =~ /^\d+/) { $f->[4] = "'$f->[4]'"; } elsif ($f->[4] =~ /^[\-]*INFINITY$/) { $f->[4] = "'$f->[4]'::$type"; } elsif ($f->[4] =~ /AT TIME ZONE/i) { $f->[4] = "($f->[4])"; } } } else { my @c = $f->[4] =~ /\./g; if ($#c >= 1) { if ($type =~ /CHAR|TEXT/i || ($was_enum && $f->[1] =~ /'/i)) { $f->[4] = "'$f->[4]'" if ($f->[4] !~ /[']/ && $f->[4] !~ /\(.*\)/ && uc($f->[4]) ne 'NULL'); } elsif ($type =~ /DATE|TIME/i) { # Case of MSSQL datetime default value to 0, must be converted to '1900-01-01 00:00:00' if ($self->{is_mssql} && $f->[4] eq "(0)") { if ($type =~ /TIME/i) { $f->[4] = '1900-01-01 00:00:00'; } else { $f->[4] = '1900-01-01'; } } # All other cases if ($f->[4] =~ /^0000-/) { if ($self->{replace_zero_date}) { $f->[4] = $self->{replace_zero_date}; } else { $f->[4] =~ s/^0000-\d+-\d+/1970-01-01/; } } if ($f->[4] =~ /^\d+/) { $f->[4] = "'$f->[4]'"; } elsif ($f->[4] =~ /^[\-]*INFINITY$/) { $f->[4] = "'$f->[4]'::$type"; } elsif ($f->[4] =~ /AT TIME ZONE/i) { $f->[4] = "($f->[4])"; } } elsif (uc($f->[4]) ne 'NULL') { $f->[4] = "'$f->[4]'"; } } elsif ($type =~ /(char|text)/i && $f->[4] !~ /^'/) { $f->[4] = "'$f->[4]'"; } } $f->[4] = 'NULL' if ($f->[4] eq "''" && $type =~ /int|double|numeric/i); $f->[4] =~ s/'((?:session|current)_[^']+)'/$1/; $sql_output .= " DEFAULT $f->[4]"; } } } } $sql_output .= ",\n"; # Replace default generated value on update by a trigger if ($f->[12] =~ /^DEFAULT_GENERATED on update (.*)/i) { $trigger_update{$f->[0]} = $1; } } if ($self->{pkey_in_create}) { $sql_output .= $self->_get_primary_keys($table, $self->{tables}{$table}{unique_key}); } $sql_output =~ s/,$//; $sql_output .= ')'; if (exists $self->{tables}{$table}{table_info}{on_commit}) { $sql_output .= ' ' . $self->{tables}{$table}{table_info}{on_commit}; } if ($self->{tables}{$table}{table_info}{partitioned} && $self->{pg_supports_partition} && !$self->{disable_partition}) { if (grep(/^$self->{partitions_list}{"\L$table\E"}{type}$/, 'HASH', 'RANGE', 'LIST') && !exists $self->{partitions_list}{"\L$table\E"}{refrtable}) { $sql_output .= " PARTITION BY " . $self->{partitions_list}{"\L$table\E"}{type} . " ("; my $expr = ''; if (exists $self->{partitions_list}{"\L$table\E"}{columns}) { my $len = $#{$self->{partitions_list}{"\L$table\E"}{columns}}; for (my $j = 0; $j <= $len; $j++) { if ($self->{partitions_list}{"\L$table\E"}{type} eq 'LIST') { $expr .= ' || ' if ($j > 0); } else { $expr .= ', ' if ($j > 0); } $expr .= $self->quote_object_name($self->{partitions_list}{"\L$table\E"}{columns}[$j]); if ($self->{partitions_list}{"\L$table\E"}{type} eq 'LIST' && $len > 0) { $expr .= '::text'; } } if ($self->{partitions_list}{"\L$table\E"}{type} eq 'LIST' && $len >= 0) { $expr = '(' . $expr . ')'; } } else { if ($self->{plsql_pgsql}) { $self->{partitions_list}{"\L$table\E"}{expression} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions_list}{"\L$table\E"}{expression}); } $expr .= $self->{partitions_list}{"\L$table\E"}{expression}; } $expr = '(' . $expr . ')' if ($expr =~ /[\(\+\-\*\%:]/ && $expr !~ /^\(.*\)$/); $sql_output .= "$expr)"; } elsif ($self->{partition_by_reference} eq 'none') { print STDERR "WARNING: unsupported partition type on table '$table'\n"; $sql_output .= " -- Unsupported partition type '" . $self->{partitions_list}{"\L$table\E"}{type} . "', please check constraint: " . $self->{partitions_list}{"\L$table\E"}{refconstraint} . "\n"; } elsif ($self->{partition_by_reference} eq 'duplicate') { $sql_output .= "DUPLICATE_EMPLACEMENT"; my $reftable = $self->{partitions_list}{"\L$table\E"}{refrtable}; $sql_output .= " PARTITION BY " . $self->{partitions_list}{"\L$reftable\E"}{type} . " ("; my $expr = ''; if (exists $self->{partitions_list}{"\L$reftable\E"}{columns}) { my $len = $#{$self->{partitions_list}{"\L$reftable\E"}{columns}}; for (my $j = 0; $j <= $len; $j++) { if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST') { $expr .= ' || ' if ($j > 0); } else { $expr .= ', ' if ($j > 0); } $expr .= $self->quote_object_name($self->{partitions_list}{"\L$reftable\E"}{columns}[$j]); if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST' && $len > 0) { $expr .= '::text'; } } if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST' && $len >= 0) { $expr = '(' . $expr . ')'; } } else { if ($self->{plsql_pgsql}) { $self->{partitions_list}{"\L$reftable\E"}{expression} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions_list}{"\L$reftable\E"}{expression}); } $expr .= $self->{partitions_list}{"\L$reftable\E"}{expression}; } $expr = '(' . $expr . ')' if ($expr =~ /[\(\+\-\*\%:]/ && $expr !~ /^\(.*\)$/); $sql_output .= "$expr)"; foreach my $k (keys %{ $self->{tables}{"$reftable"}{column_info} }) { next if (!grep(/^$k$/i, @{$self->{partitions_list}{"\L$reftable\E"}{columns}})); my $f = $self->{tables}{"$reftable"}{column_info}{$k}; $f->[2] =~ s/[^0-9\-\.]//g; # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4], 1); $type = "$f->[1], $f->[2]" if (!$type); # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}; } $sql_output =~ s/\s\)DUPLICATE_EMPLACEMENT/,\n\t\L$fname\E $type NOT NULL -- partition by reference from $table to $reftable\n)/s; } } elsif ($self->{partition_by_reference} =~ /^\d+$/) { my $reftable = $table; $sql_output .= " PARTITION BY HASH ("; my $expr = ''; if (exists $self->{partitions_list}{"\L$reftable\E"}{columns}) { my $len = $#{$self->{partitions_list}{"\L$reftable\E"}{columns}}; for (my $j = 0; $j <= $len; $j++) { if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST') { $expr .= ' || ' if ($j > 0); } else { $expr .= ', ' if ($j > 0); } $expr .= $self->quote_object_name($self->{partitions_list}{"\L$reftable\E"}{columns}[$j]); if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST' && $len > 0) { $expr .= '::text'; } } if ($self->{partitions_list}{"\L$reftable\E"}{type} eq 'LIST' && $len >= 0) { $expr = '(' . $expr . ')'; } } else { if ($self->{plsql_pgsql}) { $self->{partitions_list}{"\L$reftable\E"}{expression} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions_list}{"\L$reftable\E"}{expression}); } $expr .= $self->{partitions_list}{"\L$reftable\E"}{expression}; } $expr = '(' . $expr . ')' if ($expr =~ /[\(\+\-\*\%:]/ && $expr !~ /^\(.*\)$/); $sql_output .= "$expr)"; } } if ($obj_type =~ /\bTEMPORARY TABLE\b/) { if ($self->{tables}{$table}{table_info}{duration} eq 'SYS$TRANSACTION') { $sql_output .= ' ON COMMIT DELETE ROWS'; } elsif ($self->{tables}{$table}{table_info}{duration} eq 'SYS$SESSION') { $sql_output .= ' ON COMMIT PRESERVE ROWS'; } } if ( ($self->{type} ne 'FDW') && !$self->{oracle_fdw_data_export} && (!$self->{external_to_fdw} || (!grep(/^$table$/i, keys %{$self->{external_table}}) && !$self->{tables}{$table}{table_info}{connection})) ) { my $withoid = _make_WITH($self->{with_oid}, $self->{tables}{$table}{table_info}); if ($self->{use_tablespace} && $self->{tables}{$table}{table_info}{tablespace} && !grep(/^$self->{tables}{$table}{table_info}{tablespace}$/i, @{$self->{default_tablespaces}})) { $sql_output .= " $withoid TABLESPACE $self->{tables}{$table}{table_info}{tablespace};\n"; } else { $sql_output .= " $withoid;\n"; } } elsif ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { my $program = ''; $program = ", program '$self->{external_table}{$table}{program}'"; $sql_output .= " SERVER \L$self->{external_table}{$table}{directory}\E OPTIONS(filename '$self->{external_table}{$table}{directory_path}$self->{external_table}{$table}{location}', format 'csv', delimiter '$self->{external_table}{$table}{delimiter}'$program);\n"; } elsif ($self->{is_mysql}) { $schem = "dbname '$self->{schema}'," if ($self->{schema}); my $r_server = $self->{fdw_server}; my $r_table = $table; if ($self->{tables}{$table}{table_info}{connection} =~ /([^'\/]+)\/([^']+)/) { $r_server = $1; $r_table = $2; } $sql_output .= " SERVER $r_server OPTIONS($schem table_name '$r_table');\n"; } elsif ($self->{is_mssql}) { $schem = "$self->{schema}." if ($self->{schema}); my $r_server = $self->{fdw_server}; my $r_table = $table; if ($self->{tables}{$table}{table_info}{connection} =~ /([^'\/]+)\/([^']+)/) { $r_server = $1; $r_table = $2; } $sql_output .= " SERVER $r_server OPTIONS(table_name '$schem$r_table');\n"; } else { my $tmptb = $table; if ($self->{schema}) { $schem = "schema '$self->{schema}',"; } elsif ($tmptb =~ s/^([^\.]+)\.//) { $schem = "schema '$1',"; } $sql_output .= " SERVER $self->{fdw_server} OPTIONS($schem table '$tmptb', "; if ($self->{oracle_fdw_prefetch}) { $sql_output .= "prefetch '$self->{oracle_fdw_prefetch}', "; } $sql_output .= "readonly 'true');\n"; } } $sql_output =~ s/#ORA2PGENUM#/$enum_str/s; $sql_header .= "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";\n" if ($self->{use_uuid} && $sql_header !~ /CREATE EXTENSION .*uuid-ossp/is); # For data export from foreign table, go to next table if ($self->{oracle_fdw_data_export}) { $ib++; next; } $sql_output .= $serial_sequence; # clean up partition list foreach my $t (keys %{$self->{partitions_list}}) { my $nb = keys %{$self->{partitions_list}{$t}}; delete $self->{partitions_list}{$t} if ($nb == 0); } # Add comments on table if (!$self->{disable_comment} && $self->{tables}{$table}{table_info}{comment}) { $self->{tables}{$table}{table_info}{comment} =~ s/'/''/gs; $sql_output .= "COMMENT ON$foreign TABLE " . $self->quote_object_name($tbname) . " IS E'$self->{tables}{$table}{table_info}{comment}';\n"; } # Add comments on columns if (!$self->{disable_comment}) { foreach my $f (sort { lc($a) cmp lc($b) } keys %{$self->{tables}{$table}{column_comments}}) { next unless $self->{tables}{$table}{column_comments}{$f}; $self->{tables}{$table}{column_comments}{$f} =~ s/'/''/gs; # Change column names my $fname = $f; if (exists $self->{replaced_cols}{"\L$table\E"}{lc($fname)} && $self->{replaced_cols}{"\L$table\E"}{lc($fname)}) { $self->logit("\tReplacing column $f as " . $self->{replaced_cols}{"\L$table\E"}{lc($fname)} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$table\E"}{lc($fname)}; } $sql_output .= "COMMENT ON COLUMN " . $self->quote_object_name($tbname) . '.' . $self->quote_object_name($fname) . " IS E'" . $self->{tables}{$table}{column_comments}{$f} . "';\n"; } } # Change ownership if ($self->{force_owner} && $sql_output =~ /$tbname/is ) { my $owner = $self->{tables}{$table}{table_info}{owner}; $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); $sql_output .= "ALTER $foreign " . ($self->{tables}{$table}{table_info}{type} || 'TABLE') . " " .$self->quote_object_name($tbname) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } if (exists $self->{tables}{$table}{alter_index} && $self->{tables}{$table}{alter_index}) { foreach (@{$self->{tables}{$table}{alter_index}}) { $sql_output .= "$_;\n"; } } my $export_indexes = 1; if ((!$self->{tables}{$table}{table_info}{partitioned} || $self->{pg_version} >= 11 || $self->{disable_partition}) && $self->{type} ne 'FDW') { # Set the indexes definition my ($idx, $fts_idx) = $self->_create_indexes($table, 0, %{$self->{tables}{$table}{indexes}}); $indices .= "$idx\n" if ($idx); $fts_indices .= "$fts_idx\n" if ($fts_idx); if (!$self->{file_per_index}) { $sql_output .= $indices; $indices = ''; $sql_output .= $fts_indices; $fts_indices = ''; } # Set the unique (and primary) key definition $constraints .= $self->_create_unique_keys($table, $self->{tables}{$table}{unique_key}); # Set the check constraint definition $constraints .= $self->_create_check_constraint($table, $self->{tables}{$table}{check_constraint},$self->{tables}{$table}{field_name}, @skip_column_check); if (!$self->{file_per_constraint}) { $sql_output .= $constraints; $constraints = ''; } } if (exists $self->{tables}{$table}{alter_table} && !$self->{disable_unlogged} ) { $obj_type =~ s/UNLOGGED //; foreach (@{$self->{tables}{$table}{alter_table}}) { $sql_output .= "\nALTER $obj_type $tbname $_;\n"; } } $ib++; # Add the MySQL ON UPDATE trigger if (scalar keys %trigger_update) { my $objname = $table . '_default_'; $objname =~ s/[^a-z0-9_]//ig; $sql_output .= qq{ DROP TRIGGER IF EXISTS ${objname}_trg ON $tbname; CREATE OR REPLACE FUNCTION ${objname}_fct() RETURNS trigger AS \$\$ BEGIN }; foreach my $c (sort keys %trigger_update) { my $colname = $self->quote_object_name($c); $sql_output .= qq{ NEW.$colname = $trigger_update{$c};}; } $sql_output .= qq{ RETURN NEW; END; \$\$ LANGUAGE plpgsql; CREATE TRIGGER ${objname}_trg BEFORE UPDATE ON $tbname FOR EACH ROW EXECUTE FUNCTION ${objname}_fct(); }; } } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($ib - 1, $num_total_table, 25, '=', 'tables', 'end of table export.'), "\n"; } # When exporting data with oracle_fdw there is no more to do return $sql_output if ($self->{oracle_fdw_data_export}); if ($sequence_output && $self->{type} ne 'FDW') { my $fhdl = undef; my $qt = ''; $qt = '"' if ($self->{preserve_case}); my $fct_sequence = qq{ CREATE OR REPLACE FUNCTION ora2pg_upd_autoincrement_seq (tbname text, colname text) RETURNS VOID AS \$body\$ DECLARE query text; maxval bigint; seqname text; BEGIN query := 'SELECT max($qt' || colname || '$qt)+1 FROM $qt' || tbname || '$qt'; EXECUTE query INTO maxval; IF (maxval IS NOT NULL) THEN query := \$\$SELECT pg_get_serial_sequence ('$qt\$\$|| tbname || \$\$$qt', '\$\$ || colname || \$\$');\$\$; EXECUTE query INTO seqname; IF (seqname IS NOT NULL) THEN query := 'ALTER SEQUENCE ' || seqname || ' RESTART WITH ' || maxval; EXECUTE query; END IF; ELSE RAISE NOTICE 'Table % is empty, you must load the AUTOINCREMENT file after data import.', tbname; END IF; END; \$body\$ LANGUAGE PLPGSQL; }; $sequence_output = $fct_sequence . $sequence_output; $sequence_output .= "DROP FUNCTION ora2pg_upd_autoincrement_seq(text, text);\n"; $self->logit("Dumping DDL to restart autoincrement sequences into separate file : AUTOINCREMENT_$self->{output}\n", 1); $fhdl = $self->open_export_file("AUTOINCREMENT_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $sequence_output = $self->set_search_path() . $sequence_output; $self->dump($sql_header . $sequence_output, $fhdl); $self->close_export_file($fhdl); } if ($self->{file_per_index} && ($self->{type} ne 'FDW')) { my $fhdl = undef; $self->logit("Dumping indexes to one separate file : INDEXES_$self->{output}\n", 1); $fhdl = $self->open_export_file("INDEXES_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $indices = "-- Nothing found of type indexes\n" if (!$indices && !$self->{no_header}); $indices =~ s/\n+/\n/gs; $self->_restore_comments(\$indices); $indices = $self->set_search_path() . $indices; $self->dump($sql_header . $indices, $fhdl); $self->close_export_file($fhdl); $indices = ''; if ($fts_indices) { $fts_indices =~ s/\n+/\n/gs; my $unaccent = ''; if ($self->{use_lower_unaccent}) { $unaccent = qq{ CREATE EXTENSION IF NOT EXISTS unaccent; CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS \$\$ SELECT lower(public.unaccent('public.unaccent', \$1)); \$\$ LANGUAGE sql IMMUTABLE; }; } elsif ($self->{use_unaccent}) { $unaccent = qq{ CREATE EXTENSION IF NOT EXISTS unaccent; CREATE OR REPLACE FUNCTION unaccent_immutable(text) RETURNS text AS \$\$ SELECT public.unaccent('public.unaccent', \$1); \$\$ LANGUAGE sql IMMUTABLE; }; } # FTS TRIGGERS are exported in a separated file to be able to parallelize index creation $self->logit("Dumping triggers for FTS indexes to one separate file : FTS_INDEXES_$self->{output}\n", 1); $fhdl = $self->open_export_file("FTS_INDEXES_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->_restore_comments(\$fts_indices); $fts_indices = $self->set_search_path() . $fts_indices; $self->dump($sql_header. $unaccent . $fts_indices, $fhdl); $self->close_export_file($fhdl); $fts_indices = ''; } } # Dumping foreign key constraints my $fkeys = ''; foreach my $table (sort keys %{$self->{tables}}) { next if ($#{$self->{tables}{$table}{foreign_key}} < 0); $self->logit("Dumping RI $table...\n", 1); # Add constraint definition if ($self->{type} ne 'FDW') { my $create_all = $self->_create_foreign_keys($table); if ($create_all) { if ($self->{file_per_fkeys}) { $fkeys .= $create_all; } else { if ($self->{file_per_constraint}) { $constraints .= $create_all; } else { $sql_output .= $create_all; } } } } } if ($self->{file_per_constraint} && ($self->{type} ne 'FDW')) { my $fhdl = undef; $self->logit("Dumping constraints to one separate file : CONSTRAINTS_$self->{output}\n", 1); $fhdl = $self->open_export_file("CONSTRAINTS_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $constraints = "-- Nothing found of type constraints\n" if (!$constraints && !$self->{no_header}); $self->_restore_comments(\$constraints); $self->dump($sql_header . $constraints, $fhdl); $self->close_export_file($fhdl); $constraints = ''; } if ($fkeys) { my $fhdl = undef; $self->logit("Dumping foreign keys to one separate file : FKEYS_$self->{output}\n", 1); $fhdl = $self->open_export_file("FKEYS_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $fkeys = "-- Nothing found of type foreign keys\n" if (!$fkeys && !$self->{no_header}); $self->_restore_comments(\$fkeys); $fkeys = $self->set_search_path() . $fkeys; $self->dump($sql_header . $fkeys, $fhdl); $self->close_export_file($fhdl); $fkeys = ''; } if (!$sql_output) { $sql_output = "-- Nothing found of type TABLE\n" if (!$self->{no_header}); } else { $self->_restore_comments(\$sql_output); } $self->dump($sql_header . $sql_output); # Some virtual column have been found if ($self->{type} ne 'FDW' and scalar keys %virtual_trigger_info > 0) { my $trig_out = ''; foreach my $tb (sort keys %virtual_trigger_info) { my $tname = "virt_col_${tb}_trigger"; $tname =~ s/\./_/g; $tname = $self->quote_object_name($tname); my $fname = "fct_virt_col_${tb}_trigger"; $fname =~ s/\./_/g; $fname = $self->quote_object_name($fname); $trig_out .= "DROP TRIGGER $self->{pg_supports_ifexists} $tname ON " . $self->quote_object_name($tb) . " CASCADE;\n\n"; $trig_out .= "CREATE$self->{create_or_replace} FUNCTION $fname() RETURNS trigger AS \$BODY\$\n"; $trig_out .= "BEGIN\n"; foreach my $c (sort keys %{$virtual_trigger_info{$tb}}) { $trig_out .= "\tNEW.$c = $virtual_trigger_info{$tb}{$c};\n"; } $tb = $self->quote_object_name($tb); $trig_out .= qq{ RETURN NEW; end \$BODY\$ LANGUAGE 'plpgsql' SECURITY DEFINER; CREATE TRIGGER $tname BEFORE INSERT OR UPDATE ON $tb FOR EACH ROW EXECUTE PROCEDURE $fname(); }; } $self->_restore_comments(\$trig_out); if (!$self->{file_per_constraint}) { $self->dump($trig_out); } else { my $fhdl = undef; $self->logit("Dumping virtual column triggers to one separate file : VIRTUAL_COLUMNS_$self->{output}\n", 1); $fhdl = $self->open_export_file("VIRTUAL_COLUMNS_$self->{output}"); $self->set_binmode($fhdl) if (!$self->{compress}); $self->dump($sql_header . $trig_out, $fhdl); $self->close_export_file($fhdl); } } } =head2 _get_sql_statements Returns a string containing the PostgreSQL compatible SQL Schema definition. =cut sub _get_sql_statements { my $self = shift; # Process view if ($self->{type} eq 'VIEW') { $self->export_view(); } # Process materialized view elsif ($self->{type} eq 'MVIEW') { $self->export_mview(); } # Process grant elsif ($self->{type} eq 'GRANT') { $self->export_grant(); } # Process sequences elsif ($self->{type} eq 'SEQUENCE') { $self->export_sequence(); } # Process sequences values elsif ($self->{type} eq 'SEQUENCE_VALUES') { $self->export_sequence_values(); } # Process dblink elsif ($self->{type} eq 'DBLINK') { $self->export_dblink(); } # Process dblink elsif ($self->{type} eq 'DIRECTORY') { $self->export_directory(); } # Process triggers elsif ($self->{type} eq 'TRIGGER') { $self->export_trigger(); } # Process queries to parallelize elsif ($self->{type} eq 'LOAD') { $self->parallelize_statements(); } # Process queries only elsif ($self->{type} eq 'QUERY') { $self->translate_query(); } # Process SQL script elsif ($self->{type} eq 'SCRIPT') { $self->translate_script(); } # Process functions only elsif ($self->{type} eq 'FUNCTION') { $self->start_function_json_config($self->{type}); $self->export_function(); $self->end_function_json_config($self->{type}); } # Process procedures only elsif ($self->{type} eq 'PROCEDURE') { $self->start_function_json_config($self->{type}); $self->export_procedure(); $self->end_function_json_config($self->{type}); } # Process packages only elsif ($self->{type} eq 'PACKAGE') { $self->start_function_json_config($self->{type}); $self->export_package(); $self->end_function_json_config($self->{type}); } # Process types only elsif ($self->{type} eq 'TYPE') { $self->export_type(); } # Process TABLESPACE only elsif ($self->{type} eq 'TABLESPACE') { $self->export_tablespace(); } # Export as Kettle XML file elsif ($self->{type} eq 'KETTLE') { $self->export_kettle(); } # Process PARTITION only elsif ($self->{type} eq 'PARTITION') { $self->export_partition(); } # Process synonyms only elsif ($self->{type} eq 'SYNONYM') { $self->export_synonym(); } # Dump the database structure: tables, constraints, indexes, etc. elsif ($self->{type} eq 'TABLE' or $self->{type} eq 'FDW') { $self->export_table(); } # Extract data only elsif (($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY')) { if ($self->{oracle_fdw_data_export} && $self->{pg_dsn} && $self->{drop_foreign_schema}) { # Temporarily disable partitioning (if set) to obtain appropriate DDL for the oracle_fdw foreign table my $original_disable_partition = $self->{disable_partition}; $self->{disable_partition} = 1; my $fdw_definition = $self->export_table(); $self->{disable_partition} = $original_disable_partition; $self->{dbhdest}->do("DROP SCHEMA $self->{pg_supports_ifexists} $self->{fdw_import_schema} CASCADE") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); $self->{dbhdest}->do("CREATE SCHEMA $self->{fdw_import_schema}") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); $self->{dbhdest}->do($fdw_definition) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . ", SQL: $fdw_definition\n", 0, 1); } my $sql_output = ""; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $t0 = Benchmark->new; # Connect the Oracle database to gather information if ($self->{oracle_dsn} =~ /dbi:mysql/i) { $self->{is_mysql} = 1; } elsif ($self->{oracle_dsn} =~ /dbi:ODBC:driver=msodbcsql/i) { $self->{is_mssql} = 1; } $self->{dbh} = $self->_db_connection(); # Remove external table from data export if (scalar keys %{$self->{external_table}} ) { foreach my $table (keys %{$self->{tables}}) { if ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { delete $self->{tables}{$table}; } } } # Get current SCN to get data at a fix point in time if (!$self->{no_start_scn} && !$self->{start_scn} && !$self->{is_mysql} && !$self->{is_mssql}) { my $sth = $self->{dbh}->prepare("SELECT CURRENT_SCN FROM v\$database") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @row = $sth->fetchrow(); $self->{start_scn} = $row[0]; $sth->finish; $self->logit("Automatic storing of current SCN for table export: $self->{start_scn}\n", 1); } # Remove remote table from export, they must be exported using FDW export type foreach my $table (sort keys %{$self->{tables}}) { if ( $self->{tables}{$table}{table_info}{connection} ) { delete $self->{tables}{$table}; } } # Get partition information $self->_partitions() if (!$self->{disable_partition}); # Ordering tables by name by default my @ordered_tables = sort { $a cmp $b } keys %{$self->{tables}}; if (lc($self->{data_export_order}) eq 'size') { @ordered_tables = sort { ($self->{tables}{$b}{table_info}{num_rows} || $self->{tables}{$a}{table_info}{num_rows}) ? $self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows} : $a cmp $b } keys %{$self->{tables}}; } # User provide the ordered list of table from a file elsif (-e $self->{data_export_order}) { if (open(my $tfh, '<', $self->{data_export_order})) { @ordered_tables = (); while (my $l = <$tfh>) { chomp($l); next if (!exists $self->{tables}{$l}); push(@ordered_tables, $l); } close($tfh); } else { $self->logit("FATAL: can't read file $self->{data_export_order} for ordering table export. $!\n", 0, 1); } } # Set SQL orders that should be in the file header # (before the COPY or INSERT commands) my $first_header = "$sql_header\n"; # Add search path and constraint deferring my $search_path = $self->set_search_path(); if (!$self->{pg_dsn} && !$self->{oracle_speed}) { # Set search path if ($search_path) { $first_header .= $self->set_search_path(); } # Open transaction $first_header .= "BEGIN;\n"; # Defer all constraints if ($self->{defer_fkey}) { $first_header .= "SET CONSTRAINTS ALL DEFERRED;\n\n"; } } elsif (!$self->{oracle_speed}) { # Set search path if ($search_path) { $self->{dbhdest}->do($search_path) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } $self->{dbhdest}->do("BEGIN;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } #### Defined all SQL commands that must be executed before and after data loading if (!$self->{oracle_speed}) { my $load_file = "\n"; foreach my $table (@ordered_tables) { # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # Check that the destination table exists if ($self->{pg_dsn} && !$self->{on_error_stop}) { my $rv = $self->{dbhdest}->do("SELECT relname FROM pg_class WHERE relname = '$tmptb'"); if ($rv eq '0E0') { $self->logit("WARNING: destination table $table doesn't exists, aborting data export for this table.\n", 0); next; } } # Do not process nested table if (!$self->{is_mysql} && exists $self->{tables}{$table}{table_info}{nested} && $self->{tables}{$table}{table_info}{nested} ne 'NO') { $self->logit("WARNING: nested table $table will not be exported.\n", 1); next; } # Remove main table partition (for MySQL "SELECT * FROM emp PARTITION (p1);" is supported from 5.6) delete $self->{partitions}{$table} if (exists $self->{partitions}{$table} && $self->{is_mysql} && ($self->{db_version} =~ /^5\.[012345]/)); # Remove main table partition if we have a where clause for the table, # in this case lookup for PARTITION (p1) must not be done. delete $self->{partitions}{$table} if (exists $self->{partitions}{$table} && ($self->{global_where} or (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}))); if (-e "${dirprefix}tmp_${table}_$self->{output}") { $self->logit("Removing incomplete export file ${dirprefix}tmp_${table}_$self->{output}\n", 1); unlink("${dirprefix}tmp_${table}_$self->{output}"); } #### Set SQL commands that must be executed before data loading # Drop foreign keys if required if ($self->{drop_fkey}) { $self->logit("Dropping foreign keys of table $table...\n", 1); my @drop_all = $self->_drop_foreign_keys($table, @{$self->{tables}{$table}{foreign_key}}); foreach my $str (@drop_all) { chomp($str); next if (!$str); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $first_header .= "$str\n"; } } } # Drop indexes if required if ($self->{drop_indexes}) { $self->logit("Dropping indexes of table $table...\n", 1); my @drop_all = $self->_drop_indexes($table, %{$self->{tables}{$table}{indexes}}) . "\n"; foreach my $str (@drop_all) { chomp($str); next if (!$str); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $first_header .= "$str\n"; } } } # Disable triggers of current table if requested if ($self->{disable_triggers} && !$self->{oracle_speed}) { my $trig_type = 'USER'; $trig_type = 'ALL' if (uc($self->{disable_triggers}) eq 'ALL'); $self->logit("Disabling \L$trig_type\E triggers...\n", 1); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do("ALTER TABLE $tmptb DISABLE TRIGGER $trig_type;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $first_header .= "ALTER TABLE $tmptb DISABLE TRIGGER $trig_type;\n"; } } #### Add external data file loading if file_per_table is enable if ($self->{file_per_table} && !$self->{pg_dsn}) { my $file_name = "$dirprefix${table}_$self->{output}"; $file_name = "${table}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $load_file .= "\\i$self->{psql_relative_path} '$file_name'\n"; } # With partitioned table, load data direct from table partition if (exists $self->{partitions}{$table}) { foreach my $pos (sort {$a <=> $b} keys %{$self->{partitions}{$table}}) { my $part_name = $self->{partitions}{$table}{$pos}{name}; my $tb_name = ''; if (!exists $self->{subpartitions}{$table}{$part_name}) { $tb_name = $part_name; } $tb_name = $table . '_part' . $pos if ($self->{rename_partition}); next if ($self->{allow_partition} && !grep($_ =~ /^$part_name$/i, @{$self->{allow_partition}})); if (exists $self->{subpartitions}{$table}{$part_name}) { foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) { my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); my $sub_tb_name = $subpart; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any if ($self->{file_per_table} && !$self->{pg_dsn}) { my $file_name = "$dirprefix${table}_${sub_tb_name}_$self->{output}"; $file_name = "${table}_${sub_tb_name}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $load_file .= "\\i$self->{psql_relative_path} '$file_name'\n"; } $sub_tb_name = $tb_name . '_subpart' . $p if ($self->{rename_partition}); } # Now load content of the default partition table if ($self->{subpartitions_default}{$table}{$part_name}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}{name}$/i, @{$self->{allow_partition}})) { if ($self->{file_per_table} && !$self->{pg_dsn}) { my $part_name = $self->{subpartitions_default}{$table}{$part_name}{name}; my $file_name = "$dirprefix${table}_${part_name}_$self->{output}"; $file_name = "${table}_${part_name}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $load_file .= "\\i$self->{psql_relative_path} '$file_name'\n"; $part_name = $tb_name . '_default' if ($self->{rename_partition}); } } } } else { if ($self->{file_per_table} && !$self->{pg_dsn}) { my $part_name = $self->{partitions}{$table}{$pos}{name}; my $file_name = "$dirprefix${table}_${part_name}_$self->{output}"; $file_name = "${table}_${part_name}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $load_file .= "\\i$self->{psql_relative_path} '$file_name'\n"; } } } # Now load content of the default partition table if (exists $self->{partitions_default}{$table}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}{name}$/i, @{$self->{allow_partition}})) { if ($self->{file_per_table} && !$self->{pg_dsn}) { my $part_name = $self->{partitions_default}{$table}{name}; my $file_name = "$dirprefix${table}_${part_name}_$self->{output}"; $file_name = "${table}_${part_name}_$self->{output}" if ($self->{psql_relative_path}); $file_name =~ s/\.(gz|bz2)$//; $load_file .= "\\i$self->{psql_relative_path} '$file_name'\n"; $part_name = $table . '_part_default' if ($self->{rename_partition}); } } } } # Create temporary tables for DATADIFF if ($self->{datadiff}) { my $tmptb_del = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}); my $tmptb_ins = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); my $tmptb_upd = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_upd_suffix}); if ($self->{datadiff_work_mem}) { $first_header .= "SET work_mem TO '" . $self->{datadiff_work_mem} . "';\n"; } if ($self->{datadiff_temp_buffers}) { $first_header .= "SET temp_buffers TO '" . $self->{datadiff_temp_buffers} . "';\n"; } $first_header .= "LOCK TABLE $tmptb IN EXCLUSIVE MODE;\n"; $first_header .= "CREATE TEMPORARY TABLE $tmptb_del"; $first_header .= " (LIKE $tmptb INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES)"; $first_header .= " ON COMMIT DROP;\n"; $first_header .= "CREATE TEMPORARY TABLE $tmptb_ins"; $first_header .= " (LIKE $tmptb INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES)"; $first_header .= " ON COMMIT DROP;\n"; $first_header .= "CREATE TEMPORARY TABLE $tmptb_upd"; $first_header .= " (old $tmptb_del, new $tmptb_ins, changed_columns TEXT[])"; $first_header .= " ON COMMIT DROP;\n"; } } # When copy freeze is required, close the current transaction if ($self->{copy_freeze} && !$self->{pg_dsn}) { $first_header .= ("\nCOMMIT;\n"); } if (!$self->{pg_dsn}) { # Write header to file $self->dump($first_header); if ($self->{file_per_table}) { # Write file loader $self->dump($load_file); } } # Commit transaction if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do("COMMIT;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } } #### #### Proceed to data export #### # Set total number of rows $self->{global_rows} = 0; foreach my $table (keys %{$self->{tables}}) { if ($self->{global_where}) { if ($self->{is_mysql} && ($self->{global_where} =~ /\s+LIMIT\s+\d+,(\d+)/)) { $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); } elsif ($self->{global_where} =~ /\s+ROWNUM\s+[<=>]+\s+(\d+)/) { $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); } } elsif (exists $self->{where}{"\L$table\E"}) { if ($self->{is_mysql} && ($self->{where}{"\L$table\E"} =~ /\s+LIMIT\s+\d+,(\d+)/)) { $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); } elsif ($self->{where}{"\L$table\E"} =~ /\s+ROWNUM\s+[<=>]+\s+(\d+)/) { $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); } } $self->{global_rows} += $self->{tables}{$table}{table_info}{num_rows}; } # Open a pipe for interprocess communication my $reader = new IO::Handle; my $writer = new IO::Handle; # Fork the logger process if (!$self->{quiet} && !$self->{debug}) { if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) || ($self->{parallel_tables} > 1)) { $pipe = IO::Pipe->new($reader, $writer); $writer->autoflush(1); spawn sub { $self->multiprocess_progressbar(); }; } } $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $first_start_time = time(); my $global_count = 0; my $parallel_tables_count = 1; $self->{oracle_copies} = 1 if ($self->{parallel_tables} > 1); # Send global startup information to pipe if (defined $pipe) { $pipe->writer(); $pipe->print("GLOBAL EXPORT START TIME: $first_start_time\n"); $pipe->print("GLOBAL EXPORT ROW NUMBER: $self->{global_rows}\n"); } $self->{global_start_time} = time(); foreach my $table (@ordered_tables) { # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # Check that the destination table exists if ($self->{pg_dsn} && !$self->{on_error_stop}) { my $rv = $self->{dbhdest}->do("SELECT relname FROM pg_class WHERE relname = '$tmptb'"); if ($rv eq '0E0') { $self->logit("WARNING: destination table $table doesn't exists, aborting data export for this table.\n", 0); next; } } # Do not process nested table if (!$self->{is_mysql} && exists $self->{tables}{$table}{table_info}{nested} && $self->{tables}{$table}{table_info}{nested} ne 'NO') { $self->logit("WARNING: nested table $table will not be exported.\n", 1); next; } if ($self->{file_per_table} && !$self->{pg_dsn}) { # Do not dump data again if the file already exists next if ($self->file_exists("$dirprefix${table}_$self->{output}")); } # Set global count $global_count += $self->{tables}{$table}{table_info}{num_rows}; # Extract all column information used to determine data export. # This hash will be used in function _howto_get_data() %{$self->{colinfo}} = $self->_column_attributes($table, $self->{schema}, 'TABLE'); # Get the current SCN before getting data for this table if ($self->{cdc_ready}) { my $sth = $self->{dbh}->prepare("SELECT CURRENT_SCN FROM v\$database") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @row = $sth->fetchrow(); $self->{current_oracle_scn}{$table} = $row[0]; $sth->finish; $self->logit("Storing SCN for table $table: $self->{current_oracle_scn}{$table}\n", 1); } my $total_record = 0; if ($self->{parallel_tables} > 1) { # With partitioned table, load data direct from table partition if (!$self->{disable_parallel_partition} && !$self->{fdw_server} && exists $self->{partitions}{$table}) { foreach my $pos (sort {$self->{partitions}{$table}{$a} <=> $self->{partitions}{$table}{$b}} keys %{$self->{partitions}{$table}}) { my $part_name = $self->{partitions}{$table}{$pos}{name}; next if ($self->{allow_partition} && !grep($_ =~ /^$part_name$/i, @{$self->{allow_partition}})); if (exists $self->{subpartitions}{$table}{$part_name}) { foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) { my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); spawn sub { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $self->_export_table_data($table, $part_name, $subpart, $pos, $p, $dirprefix, $sql_header); } else { $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } # Now load content of the default subpartition table if ($self->{subpartitions_default}{$table}{$part_name}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}{name}$/i, @{$self->{allow_partition}})) { spawn sub { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $self->_export_table_data($table, $part_name, $subpart, $pos, 'default', $dirprefix, $sql_header); } else { $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } } } spawn sub { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $self->_export_table_data($table, $part_name, $subpart, $pos, undef, $dirprefix, $sql_header); } else { $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } # Now load content of the default partition table if (exists $self->{partitions_default}{$table}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}{name}$/i, @{$self->{allow_partition}})) { my $tbpart_name = $self->{partitions_default}{$table}{name}; spawn sub { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $self->_export_table_data($table, $tbpart_name, $subpart, 'default', undef, $dirprefix, $sql_header); } else { $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } } } else { spawn sub { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $self->_export_table_data($table, undef, undef, undef, undef, $dirprefix, $sql_header); } else { $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } } else { if (!$self->{fdw_server} || !$self->{pg_dsn}) { $total_record = $self->_export_table_data($table, undef, undef, undef, undef, $dirprefix, $sql_header); } else { $total_record = $self->_export_fdw_table_data($table, $dirprefix, $sql_header); } } # Display total export position if (!$self->{quiet} && !$self->{debug}) { if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1) ) { my $last_end_time = time(); my $dt = $last_end_time - $first_start_time; $dt ||= 1; my $rps = int(($total_record || $global_count) / $dt); print STDERR $self->progress_bar(($total_record || $global_count), $self->{global_rows}, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps recs/sec)"), "\r"; } } } if (!$self->{quiet} && !$self->{debug}) { if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1) ) { print "\n"; } } # Wait for all child die if ( ($self->{oracle_copies} > 1) || ($self->{parallel_tables} > 1) ) { # Wait for all child dies less the logger my $minnumchild = 1; # will not wait for progressbar process $minnumchild = 0 if ($self->{debug} || $self->{quiet}); # in debug or quiet there is no progressbar while (scalar keys %RUNNING_PIDS > $minnumchild) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { delete $RUNNING_PIDS{$kid}; } usleep(50000); } # Terminate the process logger foreach my $k (keys %RUNNING_PIDS) { kill(10, $k); %RUNNING_PIDS = (); } # Reopen a new database handler $self->{dbh}->disconnect() if (defined $self->{dbh}); if ($self->{oracle_dsn} =~ /dbi:mysql/i) { $self->{is_mysql} = 1; } elsif ($self->{oracle_dsn} =~ /dbi:ODBC:driver=msodbcsql/i) { $self->{is_mssql} = 1; } $self->{dbh} = $self->_db_connection(); } # Start a new transaction if ($self->{pg_dsn} && !$self->{oracle_speed}) { my $s = $self->{dbhdest}->do("BEGIN;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } # Remove function created to export external table if ($self->{bfile_found} eq 'text') { $self->logit("Removing function ora2pg_get_bfilename() used to retrieve path from BFILE.\n", 1); my $bfile_function = "DROP FUNCTION ora2pg_get_bfilename"; my $sth2 = $self->{dbh}->do($bfile_function); } elsif ($self->{bfile_found} eq 'efile') { $self->logit("Removing function ora2pg_get_efile() used to retrieve EFILE from BFILE.\n", 1); my $efile_function = "DROP FUNCTION ora2pg_get_efile"; my $sth2 = $self->{dbh}->do($efile_function); } elsif ($self->{bfile_found} eq 'bytea') { $self->logit("Removing function ora2pg_get_bfile() used to retrieve BFILE content.\n", 1); my $efile_function = "DROP FUNCTION ora2pg_get_bfile"; my $sth2 = $self->{dbh}->do($efile_function); } #### Set SQL commands that must be executed after data loading if (!$self->{oracle_speed}) { my $footer = ''; # When copy freeze is required, start a new the transaction if ($self->{copy_freeze} && !$self->{pg_dsn}) { $footer .= "\nBEGIN;\n"; } my (@datadiff_tbl, @datadiff_del, @datadiff_upd, @datadiff_ins); foreach my $table (@ordered_tables) { # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # Check that the destination table exists if ($self->{pg_dsn} && !$self->{on_error_stop}) { my $rv = $self->{dbhdest}->do("SELECT relname FROM pg_class WHERE relname = '$tmptb'"); if ($rv eq '0E0') { $self->logit("WARNING: destination table $table doesn't exists, aborting data export for this table.\n", 0); next; } } # Do not process nested table if (!$self->{is_mysql} && exists $self->{tables}{$table}{table_info}{nested} && $self->{tables}{$table}{table_info}{nested} ne 'NO') { $self->logit("WARNING: nested table $table will not be exported.\n", 1); next; } # DATADIFF reduction (annihilate identical deletions and insertions) and execution if ($self->{datadiff}) { my $tmptb_del = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}); my $tmptb_upd = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_upd_suffix}); my $tmptb_ins = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); my @pg_colnames_nullable = @{$self->{tables}{$table}{pg_colnames_nullable}}; my @pg_colnames_notnull = @{$self->{tables}{$table}{pg_colnames_notnull}}; my @pg_colnames_pkey = @{$self->{tables}{$table}{pg_colnames_pkey}}; # reduce by deleting matching (i.e. quasi "unchanged") entries from $tmptb_del and $tmptb_ins $footer .= "WITH del AS (SELECT t, row_number() OVER (PARTITION BY t.*) rownum, ctid FROM $tmptb_del t), "; $footer .= "ins AS (SELECT t, row_number() OVER (PARTITION BY t.*) rownum, ctid FROM $tmptb_ins t), "; $footer .= "paired AS (SELECT del.ctid ctid1, ins.ctid ctid2 FROM del JOIN ins ON del.t IS NOT DISTINCT FROM ins.t "; foreach my $col (@pg_colnames_nullable) { $footer .= "AND (((del.t).$col IS NULL AND (ins.t).$col IS NULL) OR ((del.t).$col = (ins.t).$col)) "; } foreach my $col (@pg_colnames_notnull, @pg_colnames_pkey) { $footer .= "AND ((del.t).$col = (ins.t).$col) "; } $footer .= "AND del.rownum = ins.rownum), "; $footer .= "del_del AS (DELETE FROM $tmptb_del WHERE ctid = ANY(ARRAY(SELECT ctid1 FROM paired))), "; $footer .= "del_ins AS (DELETE FROM $tmptb_ins WHERE ctid = ANY(ARRAY(SELECT ctid2 FROM paired))) "; $footer .= "SELECT 1;\n"; # convert matching delete+insert into update if configured and primary key exists if ($self->{datadiff_update_by_pkey} && $#pg_colnames_pkey >= 0) { $footer .= "WITH upd AS (SELECT old, new, old.ctid ctid1, new.ctid ctid2, ARRAY("; for my $col (@pg_colnames_notnull) { $footer .= "SELECT '$col'::TEXT WHERE old.$col <> new.$col UNION ALL "; } for my $col (@pg_colnames_nullable) { $footer .= "SELECT '$col'::TEXT WHERE old.$col <> new.$col OR ((old.$col IS NULL) <> (new.$col IS NULL)) UNION ALL "; } $footer .= "SELECT ''::TEXT WHERE FALSE) changed_columns FROM $tmptb_del old "; $footer .= "JOIN $tmptb_ins new USING (" . join(', ', @pg_colnames_pkey) . ")), "; $footer .= "del_del AS (DELETE FROM $tmptb_del WHERE ctid = ANY(ARRAY(SELECT ctid1 FROM upd))), "; $footer .= "del_ins AS (DELETE FROM $tmptb_ins WHERE ctid = ANY(ARRAY(SELECT ctid2 FROM upd))) "; $footer .= "INSERT INTO $tmptb_upd (old, new, changed_columns) SELECT old, new, changed_columns FROM upd;\n"; } # call optional function specified in config to be called before actual deletion/insertion $footer .= "SELECT " . $self->{datadiff_before} . "('" . $tmptb . "', '" . $tmptb_del . "', '" . $tmptb_upd . "', '" . $tmptb_ins . "');\n" if ($self->{datadiff_before}); # do actual delete $footer .= "WITH del AS (SELECT d.delctid FROM (SELECT t, COUNT(*) c FROM $tmptb_del t GROUP BY t) s "; $footer .= "LEFT JOIN LATERAL (SELECT ctid delctid FROM $tmptb tbl WHERE tbl IS NOT DISTINCT FROM s.t "; foreach my $col (@pg_colnames_nullable) { $footer .= "AND (((s.t).$col IS NULL AND tbl.$col IS NULL) OR ((s.t).$col = tbl.$col)) "; } foreach my $col (@pg_colnames_notnull, @pg_colnames_pkey) { $footer .= "AND ((s.t).$col = tbl.$col) "; } $footer .= "LIMIT s.c) d ON TRUE) "; $footer .= "DELETE FROM $tmptb WHERE ctid = ANY(ARRAY(SELECT delctid FROM del));\n"; # do actual update if ($self->{datadiff_update_by_pkey} && $#pg_colnames_pkey >= 0 && ($#pg_colnames_nullable >= 0 || $#pg_colnames_notnull >= 0)) { $footer .= "UPDATE $tmptb SET "; $footer .= join(', ', map { $_ . ' = (upd.new).' . $_ } @pg_colnames_notnull, @pg_colnames_nullable); $footer .= " FROM $tmptb_upd upd WHERE "; $footer .= join(' AND ', map { $_ . ' = (upd.old).' . $_ } @pg_colnames_pkey); $footer .= ";\n"; } # do actual insert $footer .= "INSERT INTO $tmptb SELECT * FROM $tmptb_ins;\n"; # call optional function specified in config to be called after actual deletion/insertion $footer .= "SELECT " . $self->{datadiff_after} . "('" . $tmptb . "', '" . $tmptb_del . "', '" . $tmptb_upd . "', '" . $tmptb_ins . "');\n" if ($self->{datadiff_after}); # push table names in array for bunch function call in the end push @datadiff_tbl, $tmptb; push @datadiff_del, $tmptb_del; push @datadiff_upd, $tmptb_upd; push @datadiff_ins, $tmptb_ins; } # disable triggers of current table if requested if ($self->{disable_triggers} && !$self->{oracle_speed}) { my $trig_type = 'USER'; $trig_type = 'ALL' if (uc($self->{disable_triggers}) eq 'ALL'); my $str = "ALTER TABLE $tmptb ENABLE TRIGGER $trig_type;"; if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $footer .= "$str\n"; } } # Recreate all foreign keys of the concerned tables if ($self->{drop_fkey} && !$self->{oracle_speed}) { my @create_all = (); $self->logit("Restoring foreign keys of table $table...\n", 1); push(@create_all, $self->_create_foreign_keys($table)); foreach my $str (@create_all) { chomp($str); next if (!$str); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . ", SQL: $str\n", 0, 1); } else { $footer .= "$str\n"; } } } # Recreate all indexes if ($self->{drop_indexes} && !$self->{oracle_speed}) { my @create_all = (); $self->logit("Restoring indexes of table $table...\n", 1); push(@create_all, $self->_create_indexes($table, 1, %{$self->{tables}{$table}{indexes}})); if ($#create_all >= 0) { foreach my $str (@create_all) { chomp($str); next if (!$str); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . ", SQL: $str\n", 0, 1); } else { $footer .= "$str\n"; } } } } } # Insert restart sequences orders if (($#ordered_tables >= 0) && !$self->{disable_sequence} && !$self->{oracle_speed}) { $self->logit("Restarting sequences\n", 1); my @restart_sequence = $self->_extract_sequence_info(); foreach my $str (@restart_sequence) { if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $footer .= "$str\n"; } } } # DATADIFF: call optional function specified in config to be called with all table names right before commit if ($self->{datadiff} && $self->{datadiff_after_all} && $#datadiff_tbl >= 0) { $footer .= "SELECT " . $self->{datadiff_after_all} . "(ARRAY['"; $footer .= join("', '", @datadiff_tbl) . "'], ARRAY['"; $footer .= join("', '", @datadiff_del) . "'], ARRAY['"; $footer .= join("', '", @datadiff_upd) . "'], ARRAY['"; $footer .= join("', '", @datadiff_ins) . "']);\n"; } } # Commit transaction if ($self->{pg_dsn} && !$self->{oracle_speed}) { my $s = $self->{dbhdest}->do("COMMIT;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } else { $footer .= "COMMIT;\n\n"; } # Recreate constraint and indexes if required $self->dump("\n$footer") if (!$self->{pg_dsn} && $footer); my $npart = 0; my $nsubpart = 0; foreach my $t (sort keys %{ $self->{partitions} }) { $npart += scalar keys %{$self->{partitions}{$t}}; } foreach my $t (sort keys %{ $self->{subpartitions_list} }) { foreach my $p (sort keys %{ $self->{subpartitions_list}{$t} }) { $nsubpart += scalar keys %{ $self->{subpartitions_list}{$t}{$p}}; } } my $t1 = Benchmark->new; my $td = timediff($t1, $t0); my $timestr = timestr($td); my $title = 'Total time to export data'; if ($self->{ora2pg_speed}) { $title = 'Total time to process data from Oracle'; } elsif ($self->{oracle_speed}) { $title = 'Total time to extract data from Oracle'; } $self->logit("$title from " . (scalar keys %{$self->{tables}}) . " tables ($npart partitions, $nsubpart sub-partitions) and $self->{global_rows} total rows: $timestr\n", 1); if ($timestr =~ /^(\d+) wallclock secs/) { my $mean = sprintf("%.2f", $self->{global_rows}/($1 || 1)); $self->logit("Speed average: $mean rows/sec\n", 1); } #### # Save SCN registered before exporting tables #### if (scalar keys %{$self->{current_oracle_scn}}) { my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); open(OUT, ">${dirprefix}TABLES_SCN.log"); print OUT "# SCN per table\n"; foreach my $t (sort keys %{$self->{current_oracle_scn}}) { print OUT "$t:$self->{current_oracle_scn}{$t}\n"; } close(OUT); } } } sub fix_function_call { my $self = shift; $self->logit("Fixing function calls in output files...\n", 0); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); return unless(open(my $tfh, '<', $dirprefix . 'temp_pass2_file.dat')); while (my $l = <$tfh>) { chomp($l); my ($pname, $fname, $file_name) = split(/:/, $l); $file_to_update{$pname}{$fname} = $file_name; } close($tfh); my $child_count = 0; # Fix call to package function in files foreach my $pname (sort keys %file_to_update ) { next if ($pname =~ /^ORA2PG_/); foreach my $fname (sort keys %{ $file_to_update{$pname} } ) { if ($self->{jobs} > 1) { while ($child_count >= $self->{jobs}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $child_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->requalify_package_functions($file_to_update{$pname}{$fname}); }; $child_count++; } else { $self->requalify_package_functions($file_to_update{$pname}{$fname}); } } } # Wait for all child end while ($child_count > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $child_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } # Requalify function call by using double quoted if necessary and by replacing # dot with an undescore when PACKAGE_AS_SCHEMA is disabled. sub requalify_package_functions { my ($self, $filename) = @_; if (open(my $fh, '<', $filename)) { $self->set_binmode($fh); my $content = ''; while (<$fh>) { $content .= $_; }; close($f); $self->requalify_function_call(\$content); if (open(my $fh, '>', $filename)) { $self->set_binmode($fh); print $fh $content; close($fh); } else { print STDERR "ERROR: requalify package functions can't write to $filename, $!\n"; return; } } else { print STDERR "ERROR: requalify package functions can't read file $filename, $!\n"; return; } } # Routine used to read input file and return content as string, # Character / is replaces by a ; and \r are removed sub read_input_file { my ($self, $file) = @_; my $content = ''; if (open(my $fin, '<', $file)) { $self->set_binmode($fin) if (_is_utf8_file( $file)); while (<$fin>) { next if /^\/$/; $content .= $_; }; close($fin); } else { die "FATAL: can't read file $file, $!\n"; } $content =~ s/[\r\n]\/([\r\n]|$)/;$2/gs; $content =~ s/\r//gs; $content =~ s/[\r\n]SHOW\s+(?:ERRORS|ERR|BTITLE|BTI|LNO|PNO|RECYCLEBIN|RECYC|RELEASE|REL|REPFOOTER|REPF|REPHEADER|REPH|SPOOL|SPOO|SGA|SQLCODE|TTITLE|TTI|USER|XQUERY|SPPARAMETERS|PARAMETERS)[^\r\n]*([\r\n]|$)/;$2/igs; if ($self->{is_mysql}) { $content =~ s/"/'/gs; $content =~ s/`/"/gs; $content =~ s/;\s*\/\//;/gs; } return $content; } sub file_exists { my ($self, $file) = @_; return 0 if ($self->{oracle_speed}); if ($self->{file_per_table} && !$self->{pg_dsn}) { if (-e "$file") { $self->logit("WARNING: Skipping dumping data to file $file, file already exists.\n", 0); return 1; } } return 0; } #### # dump table content #### sub _dump_table { my ($self, $dirprefix, $sql_header, $table, $part_name, $is_subpart, $tbpart_name, $sub_tb_name) = @_; my @cmd_head = (); my @cmd_foot = (); # Set search path my $search_path = $self->set_search_path(); if ((!$self->{truncate_table} || $self->{pg_dsn}) && $search_path) { push(@cmd_head,$search_path); } # Rename table and double-quote it if required my $tmptb = ''; if (exists $self->{replaced_tables}{"\L$table\E"} && $self->{replaced_tables}{"\L$table\E"}) { $self->logit("\tReplacing table $table as " . $self->{replaced_tables}{lc($table)} . "...\n", 1); $tmptb = $self->{replaced_tables}{lc($table)}; } # Prefix partition name with tablename, if pg_supports_partition is enabled # direct import to partition is not allowed so import to main table. if (!$self->{pg_supports_partition} && $part_name && $self->{rename_partition}) { $tmptb = $self->get_replaced_tbname($table . '_' . $part_name); } elsif (!$self->{pg_supports_partition} && $part_name) { $tmptb = $self->get_replaced_tbname($part_name || $table); } elsif ($tbpart_name) { $tmptb = $self->get_replaced_tbname($tbpart_name); $tmptb = $self->get_replaced_tbname($sub_tb_name) if ($sub_tb_name); } else { $tmptb = $self->get_replaced_tbname($table); } # Replace Tablename by temporary table for DATADIFF (data will be inserted in real table at the end) # !!! does not work correctly for partitions yet !!! if ($self->{datadiff}) { $tmptb = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); } # Build the header of the query my @tt = (); my @stt = (); my @nn = (); my $has_geometry = 0; my $has_identity = 0; $has_identity = 1 if (exists $self->{identity_info}{$table}); %{ $self->{tables}{$table}{pk_columns} } = (); # Extract column information following the Oracle position order my @fname = (); my (@pg_colnames_nullable, @pg_colnames_notnull, @pg_colnames_pkey); @{ $self->{tables}{$table}{dest_column_name} } = (); foreach my $i ( 0 .. $#{$self->{tables}{$table}{field_name}} ) { my $fieldname = ${$self->{tables}{$table}{field_name}}[$i]; next if (!$self->is_in_struct($table, $fieldname)); next if (!exists $self->{tables}{"$table"}{column_info}{"$fieldname"}); my $f = $self->{tables}{"$table"}{column_info}{"$fieldname"}; $f->[2] =~ s/\D//g; if (!$self->{enable_blob_export} && $f->[1] =~ /blob/i) { # user don't want to export blob next; } if (!$self->{enable_clob_export} && $f->[1] =~ /clob/i) { # user don't want to export clob next; } my $is_pk = $self->is_primary_key_column($table, $fieldname); # When lo_import is used we only want the PK colmuns and the BLOB if ($self->{lo_import} && $f->[1] !~ /blob/i && !$is_pk) { next; } # Get the indices and column name of the primary # for possible use for blob_to_lo data export. if ($is_pk) { $self->{tables}{$table}{pk_columns}{$i} = $fieldname; } # A virtual column must not be part of the target list next if ($f->[10] eq 'YES' and $self->{pg_supports_virtualcol}); next if (grep(/^\Q$fieldname\E$/i, @nn)); if (!$self->{preserve_case}) { push(@fname, lc($fieldname)); } else { push(@fname, $fieldname); } if ($f->[1] =~ /SDO_GEOMETRY/i) { $self->{local_type} = $self->{type} if (!$self->{local_type}); $has_geometry = 1; } my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4]); $type = "$f->[1], $f->[2]" if (!$type); if (uc($f->[1]) eq 'ENUM') { my $keyname = lc($table . '_' . $fieldname . '_t'); $f->[1] = $keyname; } # Check if this column should be replaced by a boolean following table/column name if (grep(/^\Q$fieldname\E$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { $type = 'boolean'; } # Check if this column should be replaced by a boolean following type/precision elsif (exists $self->{'replace_as_boolean'}{uc($f->[1])} && $#{ $self->{'replace_as_boolean'}{uc($f->[1])} } >= 0) { if ($self->{'replace_as_boolean'}{uc($f->[1])}[0] == $f->[5] || (!$f->[5] && $self->{'replace_as_boolean'}{uc($f->[1])}[0] == $f->[2])) { $type = 'boolean'; } } $type = $self->{'modify_type'}{lc($table)}{lc($f->[0])} if (exists $self->{'modify_type'}{lc($table)}{lc($f->[0])}); push(@stt, uc($f->[1])); push(@tt, $type); push(@nn, $fieldname); # Change column names my $colname = $f->[0]; if ($self->{replaced_cols}{lc($table)}{lc($f->[0])}) { $self->logit("\tReplacing column $f->[0] as " . $self->{replaced_cols}{lc($table)}{lc($f->[0])} . "...\n", 1); $colname = $self->{replaced_cols}{lc($table)}{lc($f->[0])}; } $colname = $self->quote_object_name($colname); if ($colname !~ /"/ && $self->is_reserved_words($colname)) { $colname = '"' . $colname . '"'; } push(@{ $self->{tables}{$table}{dest_column_name} }, $colname) if (!grep(/^\Q$colname\E$/i, @{ $self->{tables}{$table}{dest_column_name} }));; if ($self->is_primary_key_column($table, $fieldname)) { push @pg_colnames_pkey, "$colname"; } elsif ($f->[3] =~ m/^Y/) { push @pg_colnames_nullable, "$colname"; } else { push @pg_colnames_notnull, "$colname"; } } if ($self->{partition_by_reference} eq 'duplicate' && exists $self->{partitions_list}{"\L$table\E"}{refrtable}) { my $reftable = $self->{partitions_list}{"\L$table\E"}{refrtable}; foreach my $k (keys %{ $self->{tables}{"$reftable"}{column_info} }) { next if (!grep(/^\Q$k\E$/i, @{$self->{partitions_list}{"\L$reftable\E"}{columns}})); my $f = $self->{tables}{"$reftable"}{column_info}{$k}; $f->[2] =~ s/[^0-9\-\.]//g; # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4], 1); $type = "$f->[1], $f->[2]" if (!$type); # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}; } next if (grep(/^\Q$fname\E$/i, @nn)); push(@stt, uc($f->[1])); push(@tt, $type); push(@nn, $fname); # Change column names my $colname = $fname; if ($self->{replaced_cols}{lc($table)}{lc($f->[0])}) { $self->logit("\tReplacing column $f->[0] as " . $self->{replaced_cols}{lc($table)}{lc($f->[0])} . "...\n", 1); $colname = $self->{replaced_cols}{lc($table)}{lc($f->[0])}; } $colname = $self->quote_object_name($colname); if ($colname !~ /"/ && $self->is_reserved_words($colname)) { $colname = '"' . $colname . '"'; } push(@{ $self->{tables}{$table}{dest_column_name} }, $colname) if (!grep(/^\Q$colname\E$/i, @{ $self->{tables}{$table}{dest_column_name} })); } } # No column => ERROR $self->logit("FATAL: no column to export for table $table, aborting\n", 0, 1) if ($#fname < 0); $self->{tables}{$table}{pg_colnames_nullable} = \@pg_colnames_nullable; $self->{tables}{$table}{pg_colnames_notnull} = \@pg_colnames_notnull; $self->{tables}{$table}{pg_colnames_pkey} = \@pg_colnames_pkey; my $overriding_system = ''; if ($self->{pg_supports_identity}) { $overriding_system = ' OVERRIDING SYSTEM VALUE' if ($has_identity); } my $s_out = "INSERT INTO $tmptb (" . join(',', @{ $self->{tables}{$table}{dest_column_name} }); if ($self->{type} eq 'COPY') { $s_out = "\nCOPY $tmptb (" . join(',', @{ $self->{tables}{$table}{dest_column_name} }); } if ($self->{type} eq 'COPY') { $s_out .= ") FROM STDIN$self->{copy_freeze};\n"; } else { $s_out .= ")$overriding_system VALUES ("; } # Prepare statements might work in binary mode but not WKT # and INTERNAL because they use the call to ST_GeomFromText() $has_geometry = 0 if ($self->{geometry_extract_type} eq 'WKB'); # Use prepared statement in INSERT mode and only if # we are not exporting a row with a spatial column my $sprep = ''; if ($self->{pg_dsn} && !$has_geometry) { if ($self->{type} ne 'COPY') { for (my $i = 0; $i <= $#fname; $i++) { if ($stt[$i] eq 'BLOB' && $tt[$i] eq 'oid') { $s_out .= "lo_from_bytea(0, decode(?, 'hex')),"; } else { $s_out .= '?,'; } } $s_out =~ s/,$//; $s_out .= ")"; if ($self->{insert_on_conflict}) { $s_out .= " ON CONFLICT DO NOTHING"; } $sprep = $s_out; } } # Extract all data from the current table my $total_record = $self->ask_for_data($table, \@cmd_head, \@cmd_foot, $s_out, \@nn, \@tt, $sprep, \@stt, $part_name, $is_subpart); $self->{type} = $self->{local_type} if ($self->{local_type}); $self->{local_type} = ''; } #### # dump FDW table content #### sub _dump_fdw_table { my ($self, $dirprefix, $sql_header, $table, $local_dbh) = @_; my @cmd_head = (); my @cmd_foot = (); # Set search path my $search_path = $self->set_search_path(); if (!$self->{truncate_table} && $search_path) { push(@cmd_head,$search_path); } # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # Replace Tablename by temporary table for DATADIFF (data will be inserted in real table at the end) # !!! does not work correctly for partitions yet !!! if ($self->{datadiff}) { $tmptb = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); } # Build the header of the query my $col_list = ''; my $fdw_col_list = ''; my $has_geometry = 0; my $has_identity = 0; $has_identity = 1 if (exists $self->{identity_info}{$table}); # Extract column information following the Oracle position order my @fname = (); my (@pg_colnames_nullable, @pg_colnames_notnull, @pg_colnames_pkey); foreach my $i ( 0 .. $#{$self->{tables}{$table}{field_name}} ) { my $fieldname = ${$self->{tables}{$table}{field_name}}[$i]; next if (!$self->is_in_struct($table, $fieldname)); my $f = $self->{tables}{"$table"}{column_info}{"$fieldname"}; $f->[2] =~ s/\D//g; if (!$self->{enable_blob_export} && $f->[1] =~ /blob/i) { # user don't want to export blob next; } if (!$self->{enable_clob_export} && $f->[1] =~ /clob/i) { # user don't want to export clob next; } if (!$self->{preserve_case}) { push(@fname, lc($fieldname)); } else { push(@fname, $fieldname); } if ($f->[1] =~ /SDO_GEOMETRY/i) { $self->{local_type} = $self->{type} if (!$self->{local_type}); $has_geometry = 1; } my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4]); $type = "$f->[1], $f->[2]" if (!$type); # Check for boolean rewritting my $typlen = $f->[5]; $typlen ||= $f->[2]; # Check if this column should be replaced by a boolean following table/column name if (grep(/^\L$fieldname\E$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { $type = 'boolean'; } # Check if this column should be replaced by a boolean following type/precision elsif (exists $self->{'replace_as_boolean'}{uc($f->[1])} && ($self->{'replace_as_boolean'}{uc($f->[1])}[0] == $typlen)) { $type = 'boolean'; } # check if destination column type must be changed my $colname = $fieldname; $colname =~ s/["`]//g; $type = $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"} if (exists $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"}); if (uc($f->[1]) eq 'ENUM') { my $keyname = lc($table . '_' . $colname . '_t'); $type = $keyname; } # Change column names $colname = $f->[0]; if ($self->{replaced_cols}{lc($table)}{lc($f->[0])}) { $self->logit("\tReplacing column $f->[0] as " . $self->{replaced_cols}{lc($table)}{lc($f->[0])} . "...\n", 1); $colname = $self->{replaced_cols}{lc($table)}{lc($f->[0])}; } # If there is any transformation to apply replace the column name with the clause if (exists $self->{transform_value}{lc($table)}{lc($colname)}) { $fdw_col_list .= $self->{transform_value}{lc($table)}{lc($colname)} . ","; } else { # If this column is translated into boolean apply the CASE clause # except for MSSQL export through TDS_FDW because the foreign table # has already converted BIT to boolean if ($type eq 'boolean' && (uc($f->[1]) ne 'BIT' || !$self->{is_mssql} || !$self->{fdw_server})) { $fdw_col_list .= "(CASE WHEN " . $self->quote_object_name($colname) . " IS NULL THEN NULL"; my $true_list = ''; foreach my $k (keys %{$self->{ora_boolean_values}}) { if ($self->{ora_boolean_values}{$k} eq 't') { if ($f->[1] =~ /char/i) { $true_list .= " lower(" . $self->quote_object_name($colname) .") = '$k' OR"; } elsif ($k !~ /\D/) { # we only take care of numeric values $true_list .= " " . $self->quote_object_name($colname) ." = $k OR"; } } } $true_list =~ s/ OR$//; $fdw_col_list .= " WHEN ($true_list) THEN 't' ELSE 'f' END)::boolean,"; } elsif ($type eq 'oid' && $self->{blob_to_lo}) { $fdw_col_list .= "lo_from_bytea(0, " . $self->quote_object_name($colname) . "::bytea),"; } else { $fdw_col_list .= $self->quote_object_name($colname) . ","; } } $colname = $self->quote_object_name($colname); if ($colname !~ /"/ && $self->is_reserved_words($colname)) { $colname = '"' . $colname . '"'; } $col_list .= "$colname,"; if ($self->is_primary_key_column($table, $fieldname)) { push @pg_colnames_pkey, "$colname"; } elsif ($f->[3] =~ m/^Y/) { push @pg_colnames_nullable, "$colname"; } else { push @pg_colnames_notnull, "$colname"; } } $col_list =~ s/,$//; $fdw_col_list =~ s/,$//; $self->{tables}{$table}{pg_colnames_nullable} = \@pg_colnames_nullable; $self->{tables}{$table}{pg_colnames_notnull} = \@pg_colnames_notnull; $self->{tables}{$table}{pg_colnames_pkey} = \@pg_colnames_pkey; my $overriding_system = ''; if ($self->{pg_supports_identity} && $has_identity) { $overriding_system = ' OVERRIDING SYSTEM VALUE'; } my $s_out = ''; $fdwtb = $tmptb; $fdwtb = '"' . $tmptb . '"' if ($tmptb !~ /"/); if ($self->{type} eq 'INSERT') # Build INSERT statement { $s_out = "INSERT INTO $tmptb ($col_list"; $s_out .= ")$overriding_system SELECT $fdw_col_list FROM $self->{fdw_import_schema}.$fdwtb"; } if ($self->{type} eq 'COPY') # Build COPY statement { if ($self->{oracle_fdw_copy_mode} eq 'local') { $ENV{PGPASSWORD} = $self->{dbpwd}; # Need to escape the quotation marks in $fdwtb my $fdwtb_escaped = $fdwtb =~ s/"/\"/gr; $s_out = "\\copy (select $fdw_col_list from $self->{fdw_import_schema}.$fdwtb_escaped) TO PROGRAM 'psql -X -h $self->{dbhost} -p $self->{dbport} -d $self->{dbname} -U $self->{dbuser} -c \\\"\\copy $self->{schema}.$tmptb FROM STDIN " . uc($self->{oracle_fdw_copy_format}) . "\\\"' " . uc($self->{oracle_fdw_copy_format}); } if ($self->{oracle_fdw_copy_mode} eq 'server') { #$s_out = "COPY (select $fdw_col_list from $self->{fdw_import_schema}.$fdwtb) TO PROGRAM 'PGPASSWORD=$self->{dbpwd} psql -h $self->{dbhost} -p $self->{dbport} -d $self->{dbname} -U $self->{dbuser} -c \"\\copy $self->{schema}.$tmptb FROM STDIN BINARY\"' BINARY"; $s_out = "COPY (select $fdw_col_list from $self->{fdw_import_schema}.$fdwtb) TO PROGRAM 'PGPASSWORD=$self->{dbpwd} psql -h $self->{dbhost} -p $self->{dbport} -d $self->{dbname} -U $self->{dbuser} -c \"\\copy $self->{schema}.$tmptb FROM STDIN " . uc($self->{oracle_fdw_copy_format}) . "\"' " . uc($self->{oracle_fdw_copy_format}); } } $0 = "ora2pg - exporting table $self->{fdw_import_schema}.$fdwtb"; #### # Overwrite the query if REPLACE_QUERY is defined for this table #### if ($self->{replace_query}{"\L$table\E"}) { $s_out = $self->{replace_query}{"\L$table\E"}; } # Prepare statements might work in binary mode but not WKT # and INTERNAL because they use the call to ST_GeomFromText() $has_geometry = 0 if ($self->{geometry_extract_type} eq 'WKB'); # Append WHERE clause defined in the configuration file that must be applied if ($s_out !~ / WHERE /) { if (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}) { if (!$self->{is_mysql} || ($self->{where}{"\L$table\E"} !~ /\s+LIMIT\s+\d/)) { if ($self->{type} eq 'INSERT') # When using INSERT the WHERE clause needs to be added to the end { ($s_out =~ / WHERE /) ? $s_out .= ' AND ' : $s_out .= ' WHERE '; $s_out .= '(' . $self->{where}{"\L$table\E"} . ')'; } if ($self->{type} eq 'COPY') # When using COPY the WHERE clause needs to be inserted into the SELECT from FDW { # ") TO PROGRAM" needs to become "$self->{where}) TO PROGRAM" $s_out =~ s/\) TO PROGRAM/$self->{where}) TO PROGRAM/; } } else { if ($self->{type} eq 'INSERT') # When using INSERT the WHERE clause needs to be added to the end { ($s_out =~ / WHERE /) ? $s_out .= ' AND ' : $s_out .= ' WHERE '; $s_out .= $self->{where}{"\L$table\E"}; } if ($self->{type} eq 'COPY') # When using COPY the WHERE clause needs to be inserted into the SELECT from FDW { # ") TO PROGRAM" needs to become "$self->{where}) TO PROGRAM" $s_out =~ s/\) TO PROGRAM/$self->{where}) TO PROGRAM/; } } $self->logit("\tApplying WHERE clause on foreign table: " . $self->{where}{"\L$table\E"} . "\n", 1); } elsif ($self->{global_where}) { if (!$self->{is_mysql} || ($self->{global_where} !~ /\s+LIMIT\s+\d/)) { if ($self->{type} eq 'INSERT') # When using INSERT the WHERE clause needs to be added to the end { ($s_out =~ / WHERE /) ? $s_out .= ' AND ' : $s_out .= ' WHERE '; $s_out .= '(' . $self->{global_where} . ')'; } if ($self->{type} eq 'COPY') # When using COPY the WHERE clause needs to be inserted into the SELECT from FDW { # ") TO PROGRAM" needs to become "$self->{where}) TO PROGRAM" $s_out =~ s/\) TO PROGRAM/ WHERE ($self->{global_where})) TO PROGRAM/; } } else { if ($self->{type} eq 'INSERT') # When using INSERT the WHERE clause needs to be added to the end { ($s_out =~ / WHERE /) ? $s_out .= ' AND ' : $s_out .= ' WHERE '; $s_out .= $self->{global_where}; } if ($self->{type} eq 'COPY') # When using COPY the WHERE clause needs to be inserted into the SELECT from FDW { # ") TO PROGRAM" needs to become "WHERE $self->{where}) TO PROGRAM" $s_out =~ s/\) TO PROGRAM/WHERE $self->{global_where}) TO PROGRAM/; } } $self->logit("\tApplying WHERE global clause: " . $self->{global_where} . "\n", 1); } } if ( ($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"} ) { my $colpk = $self->{defined_pk}{"\L$table\E"}; if ($self->{preserve_case}) { $colpk = '"' . $colpk . '"'; } my $cond = " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; $cond = " ABS($colpk % $self->{oracle_copies}) = ?" if ($self->{is_mssql}); if ($s_out !~ s/\bWHERE\s+/WHERE $cond AND /) { if ($s_out !~ s/\b(ORDER\s+BY\s+.*)/WHERE $cond $1/) { if ($self->{type} eq 'INSERT') # When using INSERT the WHERE clause needs to be added to the end { $s_out .= " WHERE $cond"; } if ($self->{type} eq 'COPY') # When using COPY the WHERE clause needs to be inserted into the SELECT from FDW { # ") TO PROGRAM" needs to become "WHERE $cond) TO PROGRAM" $s_out =~ s/\) TO PROGRAM/ WHERE $cond) TO PROGRAM/; } } } $self->{ora_conn_count} = 0; while ($self->{ora_conn_count} < $self->{oracle_copies}) { spawn sub { if ($self->{type} eq 'INSERT') { $self->logit("Creating new connection to extract data in parallel...\n", 1); my $dbh = $local_dbh->clone(); my $search_path = $self->set_search_path(); if ($search_path) { $dbh->do($search_path) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } my $sth = $dbh->prepare($s_out) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $self->logit("Parallelizing on core #$self->{ora_conn_count} with query: $s_out\n", 1); $self->logit("Exporting foreign table data for $table, #$self->{ora_conn_count}\n", 1); $sth->execute($self->{ora_conn_count}) or $self->logit("FATAL: " . $dbh->errstr . ", SQL: $s_out\n", 0, 1); $sth->finish(); if (defined $pipe) { my $t_time = time(); $pipe->print("TABLE EXPORT ENDED: $table, end: $t_time, rows $self->{tables}{$table}{table_info}{num_rows}\n"); } $dbh->disconnect() if ($dbh); } if ($self->{type} eq 'COPY' and $self->{oracle_fdw_copy_mode} eq 'local') { # Need to replace the "?" in $s_out with the relevant integer ("$self->{ora_conn_count}") $s_out =~ s/\?/$self->{ora_conn_count}/; $self->logit("Parallelizing on core #$self->{ora_conn_count} using psql command: $s_out\n", 1); $ENV{PGPASSWORD} = $self->{dbpwd}; my $psql_cmd = "psql -X -h $self->{dbhost} -p $self->{dbport} -d $self->{dbname} -U $self->{dbuser} -c \"$s_out\""; $self->logit("Exporting foreign table data for $table, #$self->{ora_conn_count}\n", 1); my $cmd_output = `$psql_cmd` or $self->logit("FATAL: " . $cmd_output . "\n", 0, 1); if (defined $pipe) { my $t_time = time(); $pipe->print("TABLE EXPORT ENDED: $table, end: $t_time, rows $self->{tables}{$table}{table_info}{num_rows}\n"); } } if ($self->{type} eq 'COPY' and $self->{oracle_fdw_copy_mode} eq 'server') { $self->logit("Creating new connection to extract data in parallel...\n", 1); my $dbh = $local_dbh->clone(); my $search_path = $self->set_search_path(); if ($search_path) { $dbh->do($search_path) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } my $sth = $dbh->prepare($s_out) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my $s_out_no_password = $s_out =~ s/PGPASSWORD=[^\s]+\s/PGPASSWORD=********** /r; $self->logit("Parallelizing on core #$self->{ora_conn_count} with query: $s_out_no_password\n", 1); $self->logit("Exporting foreign table data for $table, #$self->{ora_conn_count}\n", 1); $sth->execute($self->{ora_conn_count}) or $self->logit("FATAL: " . $dbh->errstr . ", SQL: $s_out\n", 0, 1); $sth->finish(); if (defined $pipe) { my $t_time = time(); $pipe->print("TABLE EXPORT ENDED: $table, end: $t_time, rows $self->{tables}{$table}{table_info}{num_rows}\n"); } $dbh->disconnect() if ($dbh); } }; $self->{ora_conn_count}++; } # Wait for oracle connection terminaison while ($self->{ora_conn_count} > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{ora_conn_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } else { if ($self->{type} eq 'INSERT') { if ($search_path) { $local_dbh->do($search_path) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); } $self->logit("Exporting foreign table data for $table using query: $s_out\n", 1); $local_dbh->do($s_out) or $self->logit("ERROR: " . $local_dbh->errstr . ", SQL: $s_out\n", 0); } if ($self->{type} eq 'COPY' and $self->{oracle_fdw_copy_mode} eq 'local') { $ENV{PGPASSWORD} = $self->{dbpwd}; my $psql_cmd = "psql -X -h $self->{dbhost} -p $self->{dbport} -d $self->{dbname} -U $self->{dbuser} -c \"$s_out\""; $self->logit("Exporting foreign table data for $table using psql command: $s_out\n", 1); my $cmd_output = `$psql_cmd` or $self->logit("FATAL: " . $cmd_output . "\n", 0, 1); } if ($self->{type} eq 'COPY' and $self->{oracle_fdw_copy_mode} eq 'server') { $self->logit("Exporting foreign table data for $table using query: $s_out\n", 1); $local_dbh->do($s_out) or $self->logit("ERROR: " . $local_dbh->errstr . ", SQL: $s_out\n", 0); } } $self->{type} = $self->{local_type} if ($self->{local_type}); $self->{local_type} = ''; } sub exclude_mviews { my ($self, $cols) = @_; my $sql = " AND ($cols) NOT IN (SELECT OWNER, TABLE_NAME FROM $self->{prefix}_OBJECT_TABLES)"; $sql .= " AND ($cols) NOT IN (SELECT OWNER, MVIEW_NAME FROM $self->{prefix}_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM $self->{prefix}_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); return $sql; } =head2 _column_comments This function return comments associated to columns =cut sub _column_comments { my ($self, $table) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_column_comments($self, $table); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_column_comments($self, $table); } else { return Ora2Pg::Oracle::_column_comments($self, $table); } } sub get_indexname { my ($self, $table, $idx, @collist) = @_; my $schm = ''; my $idxname = ''; if ($idx =~ /^([^\.]+)\.(.*)$/) { $schm = $1; $idxname = $2; } else { $idxname = $idx; } if ($self->{indexes_renaming}) { if ($table =~ /^([^\.]+)\.(.*)$/) { $schm = $1; $idxname = $2; } else { $idxname = $table; } $idxname =~ s/"//g; # Remove double quote, DESC and parenthesys map { s/"//g; s/.*\(([^\)]+)\).*/$1/; s/\s+DESC//i; s/::.*//; } @collist; $idxname .= '_' . join('_', @collist); $idxname =~ s/\s+//g; if ($self->{indexes_suffix}) { $idxname = substr($idxname,0,59); } else { $idxname = substr($idxname,0,63); } } # Remove non alphanumeric character #$idxname =~ s/[^a-z0-9_]+//ig; $idxname = $self->quote_object_name("$idxname$self->{indexes_suffix}"); return $idxname; } =head2 _create_indexes This function return SQL code to create indexes of a table and triggers to create for FTS indexes. - $indexonly mean no FTS index output =cut sub _create_indexes { my ($self, $table, $indexonly, %indexes) = @_; my $tbsaved = $table; # The %indexes hash can be passed from table or materialized views definition my $objtyp = 'tables'; if (!exists $self->{tables}{$tbsaved} && exists $self->{materialized_views}{$tbsaved}) { $objtyp = 'materialized_views'; } my %pkcollist = (); # Save the list of column for PK to check unique index that must be removed foreach my $consname (keys %{$self->{$objtyp}{$tbsaved}{unique_key}}) { next if ($self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{type} ne 'P'); my @conscols = grep(!/^\d+$/, @{$self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{columns}}); # save the list of column for PK to check unique index that must be removed $pkcollist{$tbsaved} = join(", ", @conscols); } $pkcollist{$tbsaved} =~ s/\s+/ /g; $table = $self->get_replaced_tbname($table); my @out = (); my @fts_out = (); my $has_to_char = 0; # Set the index definition foreach my $idx (sort keys %indexes) { # Remove cols than have only digit as name @{$indexes{$idx}} = grep(!/^\d+$/, @{$indexes{$idx}}); # Cluster, bitmap join, reversed and IOT indexes will not be exported at all # Hash indexes will be exported as btree if PG < 10 next if ($self->{tables}{$tbsaved}{idx_type}{$idx}{type} =~ /JOIN|IOT|CLUSTER|REV/i); if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { map { s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/i } @{$indexes{$idx}}; } } my @strings = (); my $i = 0; for (my $j = 0; $j <= $#{$indexes{$idx}}; $j++) { $indexes{$idx}->[$j] =~ s/''/%%ESCAPED_STRING%%/g; while ($indexes{$idx}->[$j] =~ s/'([^']+)'/%%string$i%%/) { push(@strings, $1); $i++; } if ($self->{plsql_pgsql}) { $indexes{$idx}->[$j] = Ora2Pg::PLSQL::convert_plsql_code($self, $indexes{$idx}->[$j], @strings); } $indexes{$idx}->[$j] =~ s/%%ESCAPED_STRING%%/''/ig; $has_to_char = 1 if ($indexes{$idx}->[$j] =~ s/TO_CHAR\s*\(/immutable_to_char\(/ig); } # Add index opclass if required and type allow it my %opclass_type = (); if ($self->{use_index_opclass}) { my $i = 0; for (my $j = 0; $j <= $#{$indexes{$idx}}; $j++) { if (exists $self->{$objtyp}{$tbsaved}{column_info}{uc($indexes{$idx}->[$j])}) { my $d = $self->{$objtyp}{$tbsaved}{column_info}{uc($indexes{$idx}->[$j])}; $d->[2] =~ s/\D//g; if ( (($self->{use_index_opclass} == 1) || ($self->{use_index_opclass} <= $d->[2])) && ($d->[1] =~ /VARCHAR/)) { my $typ = $self->_sql_type($d->[1], $d->[2], $d->[5], $d->[6], $f->[4]); $typ =~ s/\(.*//; if ($typ =~ /varchar/) { $typ = ' varchar_pattern_ops'; } elsif ($typ =~ /text/) { $typ = ' text_pattern_ops'; } elsif ($typ =~ /char/) { $typ = ' bpchar_pattern_ops'; } $opclass_type{$indexes{$idx}->[$j]} = "$indexes{$idx}->[$j] $typ"; } } } } # Add parentheses to index column definition when a space or arithmetic operators are found if (!$self->{input_file}) { for ($i = 0; $i <= $#{$indexes{$idx}}; $i++) { if ( $indexes{$idx}->[$i] =~ /[\s\-\+\/\*]/ && $indexes{$idx}->[$i] !~ /^[^\.\s]+\s+(ASC|DESC)$/i && $indexes{$idx}->[$i] !~ /\s+collate\s+/i ) { $indexes{$idx}->[$i] = '(' . $indexes{$idx}->[$i] . ')'; } $indexes{$idx}->[$i] =~ s/"//g; } } else { for ($i = 0; $i <= $#{$indexes{$idx}}; $i++) { my @tmp_col = split(/\s*,\s*/, $indexes{$idx}->[$i]); for (my $j = 0; $j <= $#tmp_col; $j++) { if ( $tmp_col[$j] =~ /[\s\-\+\/\*]/ && $tmp_col[$j] !~ /^[^\.\s]+\s+(ASC|DESC)$/i && $tmp_col[$j] !~ /\s+collate\s+/i ) { $tmp_col[$j] = '(' . $tmp_col[$j] . ')'; } } $indexes{$idx}->[$i] = join(', ', @tmp_col); } } my $columns = ''; foreach my $s (@{$indexes{$idx}}) { $s = '"' . $s . '"' if ($self->is_reserved_words($s)); if ($s =~ /\|\|/i) { $columns .= '(' . $s . ')'; } else { if ($s =~ /^CASE\s+.*END/i) { $s = "($s)"; } $columns .= ((exists $opclass_type{$s}) ? $opclass_type{$s} : $s) . ", "; } # Add double quotes on column name if PRESERVE_CASE is enabled foreach my $c (keys %{$self->{tables}{$tbsaved}{column_info}}) { $columns =~ s/\b$c\b/"$c"/g if ($self->{preserve_case} && $columns !~ /"$c"/); } } $columns =~ s/, $//s; $columns =~ s/\s+/ /gs; my $colscompare = $columns; $colscompare =~ s/"//g; $colscompare =~ s/ //g; my $columnlist = ''; my $skip_index_creation = 0; my %pk_hist = (); foreach my $consname (keys %{$self->{$objtyp}{$tbsaved}{unique_key}}) { my $constype = $self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{type}; next if (($constype ne 'P') && ($constype ne 'U')); my @conscols = grep(!/^\d+$/, @{$self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{columns}}); for ($i = 0; $i <= $#conscols; $i++) { # Change column names if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}) { $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; } } $columnlist = join(',', @conscols); $columnlist =~ s/"//gs; $columnlist =~ s/\s+//gs; if ($constype eq 'P') { $pk_hist{$table} = $columnlist; } if (lc($columnlist) eq lc($colscompare)) { $skip_index_creation = 1; last; } } # Do not create the index if there already a constraint on the same column list # or there a primary key defined on the same columns as a unique index, in both cases # the index will be automatically created by PostgreSQL at constraint import time. if (!$skip_index_creation) { my $unique = ''; $unique = ' UNIQUE' if ($self->{$objtyp}{$tbsaved}{uniqueness}{$idx} eq 'UNIQUE'); my $str = ''; my $fts_str = ''; my $concurrently = ''; if ($self->{$objtyp}{$tbsaved}{concurrently}{$idx}) { $concurrently = ' CONCURRENTLY'; } $columns = lc($columns) if (!$self->{preserve_case}); next if ( lc($columns) eq lc($pkcollist{$tbsaved}) ); for ($i = 0; $i <= $#strings; $i++) { $columns =~ s/\%\%string$i\%\%/'$strings[$i]'/; } # Replace call of schema.package.function() into package.function() $columns =~ s/\b[^\s\.]+\.([^\s\.]+\.[^\s\.]+)\s*\(/$1\(/is; # Do not create indexes if they are already defined as constraints if ($self->{type} eq 'TABLE') { my $col_list = $columns; $col_list =~ s/"//g; $col_list =~ s/, /,/g; next if (exists $pk_hist{$table} && uc($pk_hist{$table}) eq uc($col_list)); } my $idxname = $self->get_indexname($table, $idx, @{$indexes{$idx}}); $str .= "DROP INDEX $self->{pg_supports_ifexists} $idxname;\n" if ($self->{drop_if_exists}); my $tb = $self->quote_object_name($table); if ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /SPATIAL_INDEX/) { $str .= "CREATE INDEX$concurrently " . $idxname . " ON $tb USING gist($columns)"; } elsif ($self->{bitmap_as_gin} && ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} eq 'BITMAP' || $self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type} eq 'BITMAP')) { $str .= "CREATE INDEX$concurrently " . $idxname . " ON $tb USING gin($columns)"; } elsif ( ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /CTXCAT/) || ($self->{context_as_trgm} && ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/)) ) { # use pg_trgm my @cols = split(/\s*,\s*/, $columns); map { s/^(.*)$/unaccent_immutable($1)/; } @cols if ($self->{use_unaccent}); $columns = join(" gin_trgm_ops, ", @cols); $columns .= " gin_trgm_ops"; $str .= "CREATE INDEX$concurrently " . $idxname . " ON $tb USING gin($columns)"; } elsif (($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/) && $self->{fts_index_only}) { my $stemmer = $self->{fts_config} || lc($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{stemmer}) || 'pg_catalog.english'; my $dico = $stemmer; $dico =~ s/^pg_catalog\.//; if ($self->{use_unaccent}) { $dico =~ s/^(..).*/$1/; if ($fts_str !~ /CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);/s) { $fts_str .= "CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);\n"; $stemmer =~ s/pg_catalog\.//; $fts_str .= "ALTER TEXT SEARCH CONFIGURATION $dico ALTER MAPPING FOR hword, hword_part, word WITH unaccent, ${stemmer}_stem;\n\n"; } } # use function-based index" my @cols = split(/\s*,\s*/, $columns); $columns = "to_tsvector('$dico', " . join("||' '||", @cols) . ")"; $fts_str .= "CREATE INDEX$concurrently " . $idxname . " ON $tb USING gin($columns);\n"; } elsif (($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/) && !$self->{fts_index_only}) { # use Full text search, then create dedicated column and trigger before the index. map { s/"//g; } @{$indexes{$idx}}; my $newcolname = join('_', @{$indexes{$idx}}); $fts_str .= "\n-- Append the FTS column to the table\n"; $fts_str .= "\nALTER TABLE $tb ADD COLUMN tsv_" . substr($newcolname,0,59) . " tsvector;\n"; my $fctname = "tsv_${table}_" . substr($newcolname,0,59-(length($table)+1)); my $trig_name = "trig_tsv_${table}_" . substr($newcolname,0,54-(length($table)+1)); my $contruct_vector = ''; my $update_vector = ''; my $weight = 'A'; my $stemmer = $self->{fts_config} || lc($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{stemmer}) || 'pg_catalog.english'; my $dico = $stemmer; $dico =~ s/^pg_catalog\.//; if ($self->{use_unaccent}) { $dico =~ s/^(..).*/$1/; if ($fts_str !~ /CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);/s) { $fts_str .= "CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);\n"; $stemmer =~ s/pg_catalog\.//; $fts_str .= "ALTER TEXT SEARCH CONFIGURATION $dico ALTER MAPPING FOR hword, hword_part, word WITH unaccent, ${stemmer}_stem;\n\n"; } } if ($#{$indexes{$idx}} > 0) { foreach my $col (@{$indexes{$idx}}) { $contruct_vector .= "\t\tsetweight(to_tsvector('$dico', coalesce(new.$col,'')), '$weight') ||\n"; $update_vector .= " setweight(to_tsvector('$dico', coalesce($col,'')), '$weight') ||"; $weight++; } $contruct_vector =~ s/\|\|$/;/s; $update_vector =~ s/\|\|$/;/s; } else { $contruct_vector = "\t\tto_tsvector('$dico', coalesce(new.$indexes{$idx}->[0],''))\n"; $update_vector = " to_tsvector('$dico', coalesce($indexes{$idx}->[0],''))"; } $fts_str .= qq{ -- When the data migration is done without trigger, create tsvector data for all the existing records UPDATE $tb SET tsv_$newcolname = $update_vector -- Trigger used to keep fts field up to date CREATE FUNCTION $fctname() RETURNS trigger AS \$\$ BEGIN IF TG_OP = 'INSERT' OR new.$newcolname != old.$newcolname THEN new.tsv_$newcolname := $contruct_vector END IF; return new; END \$\$ LANGUAGE plpgsql; CREATE TRIGGER $trig_name BEFORE INSERT OR UPDATE ON $tb FOR EACH ROW EXECUTE PROCEDURE $fctname(); } if (!$indexonly); if ($objtyp eq 'tables') { $str .= "CREATE$unique INDEX$concurrently " . $idxname . " ON $table USING gin(tsv_$newcolname)"; } else { $fts_str .= "CREATE$unique INDEX$concurrently " . $idxname . " ON $table USING gin(tsv_$newcolname)"; } } elsif ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type} =~ /DOMAIN/i && $self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} !~ /SPATIAL_INDEX/) { $str .= "-- Was declared as DOMAIN index, please check for FTS adaptation if require\n"; $str .= "-- CREATE$unique INDEX$concurrently " . $idxname . " ON $table ($columns)"; } else { $str .= "CREATE$unique INDEX$concurrently " . $idxname . " ON $table ($columns)"; } if ($#{$self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_include}} >= 0) { $str .= " INCLUDE (" . join(', ', @{$self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_include}}) . ')'; } if ($self->{use_tablespace} && $self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx} && !grep(/^$self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx}$/i, @{$self->{default_tablespaces}})) { $str .= " TABLESPACE $self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx}"; } if ($str) { $str .= ";"; push(@out, $str); } push(@fts_out, $fts_str) if ($fts_str); } } if ($has_to_char) { unshift(@out, qq{ -- Function used in indexes must be immutable, use immutable_to_char() instead of to_char() CREATE OR REPLACE FUNCTION immutable_to_char(timestamp, fmt text) RETURNS text AS \$\$ SELECT to_char(\$1, \$2); \$\$ LANGUAGE sql immutable; }); } return $indexonly ? (@out,@fts_out) : (join("\n", @out), join("\n", @fts_out)); } =head2 _drop_indexes This function return SQL code to drop indexes of a table =cut sub _drop_indexes { my ($self, $table, %indexes) = @_; my $tbsaved = $table; $table = $self->get_replaced_tbname($table); my @out = (); # Set the index definition foreach my $idx (keys %indexes) { # Cluster, bitmap join, reversed and IOT indexes will not be exported at all next if ($self->{tables}{$tbsaved}{idx_type}{$idx}{type} =~ /JOIN|IOT|CLUSTER|REV/i); if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { map { s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/i } @{$indexes{$idx}}; } } map { if ($_ !~ /\(.*\)/) { $_ =~ s/(\s+.*)//; # DESC or ASC $_ = $self->quote_object_name($_); $_ .= $1; } } @{$indexes{$idx}}; my $columns = ''; foreach my $s (@{$indexes{$idx}}) { if ($s =~ /\|\|/) { $columns .= '(' . $s . ')'; } else { $columns .= ((exists $opclass_type{$s}) ? $opclass_type{$s} : $s) . ", "; } # Add double quotes on column name if PRESERVE_CASE is enabled foreach my $c (keys %{$self->{tables}{$tbsaved}{column_info}}) { $columns =~ s/\b$c\b/"$c"/ if ($self->{preserve_case} && $columns !~ /"$c"/); } } $columns =~ s/, $//s; $columns =~ s/\s+//gs; my $colscompare = $columns; $colscompare =~ s/"//gs; my $columnlist = ''; my $skip_index_creation = 0; my %pk_hist = (); foreach my $consname (keys %{$self->{tables}{$tbsaved}{unique_key}}) { my $constype = $self->{tables}{$tbsaved}{unique_key}->{$consname}{type}; next if (($constype ne 'P') && ($constype ne 'U')); my @conscols = grep(!/^\d+$/, @{$self->{tables}{$tbsaved}{unique_key}->{$consname}{columns}}); for ($i = 0; $i <= $#conscols; $i++) { # Change column names if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}) { $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; } } $columnlist = join(',', @conscols); $columnlist =~ s/"//gs; $columnlist =~ s/\s+//gs; if ($constype eq 'P') { $pk_hist{$table} = $columnlist; } if (lc($columnlist) eq lc($colscompare)) { $skip_index_creation = 1; last; } } # Do not create the index if there already a constraint on the same column list # the index will be automatically created by PostgreSQL at constraint import time. if (!$skip_index_creation) { # Cluster, bitmap join, reversed and IOT indexes will not be exported at all # Hash indexes will be exported as btree if PG < 10 next if ($self->{tables}{$tbsaved}{idx_type}{$idx}{type} =~ /JOIN|IOT|CLUSTER|REV/i); if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { map { s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/i } @{$indexes{$idx}}; } } for (my $j = 0; $j <= $#{$indexes{$idx}}; $j++) { $indexes{$idx}->[$j] =~ s/''/%%ESCAPED_STRING%%/g; my @strings = (); my $i = 0; while ($indexes{$idx}->[$j] =~ s/'([^']+)'/%%string$i%%/) { push(@strings, $1); $i++; } if ($self->{plsql_pgsql}) { $indexes{$idx}->[$j] = Ora2Pg::PLSQL::convert_plsql_code($self, $indexes{$idx}->[$j], @strings); } $indexes{$idx}->[$j] =~ s/%%ESCAPED_STRING%%/''/ig; for ($i = 0; $i <= $#strings; $i++) { $indexes{$idx}->[$j] =~ s/\%\%string$i\%\%/'$strings[$i]'/; } } my $idxname = $self->get_indexname($table, $idx, @{$indexes{$idx}}); if ($self->{tables}{$table}{idx_type}{$idx}{type} =~ /DOMAIN/i && $self->{tables}{$table}{idx_type}{$idx}{type_name} !~ /SPATIAL_INDEX/) { push(@out, "-- Declared as DOMAIN index, uncomment line below if it must be removed"); push(@out, "-- DROP INDEX $self->{pg_supports_ifexists} $idxname\L$self->{indexes_suffix}\E;"); } else { push(@out, "DROP INDEX $self->{pg_supports_ifexists} $idxname\L$self->{indexes_suffix}\E;"); } } } return wantarray ? @out : join("\n", @out); } =head2 _exportable_indexes This function return the indexes that will be exported =cut sub _exportable_indexes { my ($self, $table, %indexes) = @_; my @out = (); # Set the index definition foreach my $idx (keys %indexes) { map { if ($_ !~ /\(.*\)/) { s/^/"/; s/$/"/; } } @{$indexes{$idx}}; map { s/"//gs } @{$indexes{$idx}}; my $columns = join(',', @{$indexes{$idx}}); my $colscompare = $columns; my $columnlist = ''; my $skip_index_creation = 0; foreach my $consname (keys %{$self->{tables}{$table}{unique_key}}) { my $constype = $self->{tables}{$table}{unique_key}->{$consname}{type}; next if (($constype ne 'P') && ($constype ne 'U')); my @conscols = @{$self->{tables}{$table}{unique_key}->{$consname}{columns}}; $columnlist = join(',', @conscols); $columnlist =~ s/"//gs; if (lc($columnlist) eq lc($colscompare)) { $skip_index_creation = 1; last; } } # The index will not be created if (!$skip_index_creation) { push(@out, $idx); } } return @out; } =head2 is_primary_key_column This function return 1 when the specified column is a primary key =cut sub is_primary_key_column { my ($self, $table, $col) = @_; # Set the unique (and primary) key definition foreach my $consname (keys %{ $self->{tables}{$table}{unique_key} }) { next if ($self->{tables}{$table}{unique_key}->{$consname}{type} ne 'P'); my @conscols = @{$self->{tables}{$table}{unique_key}->{$consname}{columns}}; for (my $i = 0; $i <= $#conscols; $i++) { if (lc($conscols[$i]) eq lc($col)) { return 1; } } } return 0; } =head2 _get_primary_keys This function return SQL code to add primary keys of a create table definition =cut sub _get_primary_keys { my ($self, $table, $unique_key) = @_; my $out = ''; # Set the unique (and primary) key definition foreach my $consname (keys %$unique_key) { next if ($self->{pkey_in_create} && ($unique_key->{$consname}{type} ne 'P')); my $constype = $unique_key->{$consname}{type}; my $constgen = $unique_key->{$consname}{generated}; my $index_name = $unique_key->{$consname}{index_name}; my @conscols = @{$unique_key->{$consname}{columns}}; my %constypenames = ('U' => 'UNIQUE', 'P' => 'PRIMARY KEY'); my $constypename = $constypenames{$constype}; for (my $i = 0; $i <= $#conscols; $i++) { # Change column names if (exists $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"}) { $conscols[$i] = $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"}; } } map { $_ = $self->quote_object_name($_) } @conscols; my $columnlist = join(',', @conscols); if ($columnlist) { if ($self->{pkey_in_create}) { if (!$self->{keep_pkey_names} || ($constgen eq 'GENERATED NAME')) { $out .= "\tPRIMARY KEY ($columnlist)"; } else { $out .= "\tCONSTRAINT " . $self->quote_object_name($consname) . " PRIMARY KEY ($columnlist)"; } if ($self->{use_tablespace} && $self->{tables}{$table}{idx_tbsp}{$index_name} && !grep(/^$self->{tables}{$table}{idx_tbsp}{$index_name}$/i, @{$self->{default_tablespaces}})) { $out .= " USING INDEX TABLESPACE " . $self->quote_object_name($self->{tables}{$table}{idx_tbsp}{$index_name}); } $out .= ",\n"; } } } $out =~ s/,$//s; return $out; } =head2 _create_unique_keys This function return SQL code to create unique and primary keys of a table =cut sub _create_unique_keys { my ($self, $table, $unique_key, $partition) = @_; my $out = ''; my $tbsaved = $table; $table = $self->get_replaced_tbname($table); # Set the unique (and primary) key definition foreach my $consname (keys %$unique_key) { next if ($self->{pkey_in_create} && ($unique_key->{$consname}{type} eq 'P')); my $constype = $unique_key->{$consname}{type}; my $constgen = $unique_key->{$consname}{generated}; my $index_name = $unique_key->{$consname}{index_name}; my $deferrable = $unique_key->{$consname}{deferrable}; my $deferred = $unique_key->{$consname}{deferred}; my @conscols = @{$unique_key->{$consname}{columns}}; # Exclude unique index used in PK when column list is the same next if (($constype eq 'U') && exists $pkcollist{$table} && ($pkcollist{$table} eq join(",", @conscols))); my %constypenames = ('U' => 'UNIQUE', 'P' => 'PRIMARY KEY'); my $constypename = $constypenames{$constype}; for (my $i = 0; $i <= $#conscols; $i++) { # Change column names if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\L"}{"\L$conscols[$i]\E"}) { $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; } } # Add the partition column if it is not is the PK if (!$self->{disable_partition} && ($constype eq 'P' || $constype eq 'U') && exists $self->{partitions_list}{"\L$tbsaved\E"}) { for (my $j = 0; $j <= $#{$self->{partitions_list}{"\L$tbsaved\E"}{columns}}; $j++) { push(@conscols, $self->{partitions_list}{"\L$tbsaved\E"}{columns}[$j]) if (!grep(/^$self->{partitions_list}{"\L$tbsaved\E"}{columns}[$j]$/i, @conscols)); } if ($partition) { for (my $j = 0; $j <= $#{$self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$partition\E"}{columns}}; $j++) { push(@conscols, $self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$partition\E"}{columns}[$j]) if (!grep(/^$self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$partition\E"}{columns}[$j]$/i, @conscols)); } } else { foreach my $part (keys %{$self->{subpartitions_list}{"\L$tbsaved\E"}}) { for (my $j = 0; $j <= $#{$self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$part\E"}{columns}}; $j++) { push(@conscols, $self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$part\E"}{columns}[$j]) if (!grep(/^$self->{subpartitions_list}{"\L$tbsaved\E"}{"\L$part\E"}{columns}[$j]$/i, @conscols)); } } } } map { $_ = $self->quote_object_name($_) } @conscols; my $reftable = $table; $reftable = $self->{partitions_list}{"\L$table\E"}{refrtable} if (exists $self->{partitions_list}{"\L$table\E"}{refrtable}); foreach my $k (keys %{ $self->{tables}{"$reftable"}{column_info} }) { next if (!grep(/^$k$/i, @{$self->{partitions_list}{"\L$reftable\E"}{columns}})); my $f = $self->{tables}{"$reftable"}{column_info}{$k}; $f->[2] =~ s/[^0-9\-\.]//g; # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}; } push(@conscols, $self->quote_object_name($fname)); } my $columnlist = join(',', @conscols); if ($columnlist) { if (!$self->{keep_pkey_names} || ($constgen eq 'GENERATED NAME')) { $str .= "ALTER TABLE $table DROP $constypename;\n" if ($self->{drop_if_exists}); $out .= "ALTER TABLE $table ADD $constypename ($columnlist)"; } else { $str .= "ALTER TABLE $table DROP CONSTRAINT $self->{pg_supports_ifexists} " . $self->quote_object_name($consname) . ";\n" if ($self->{drop_if_exists}); $out .= "ALTER TABLE $table ADD CONSTRAINT " . $self->quote_object_name($consname) . " $constypename ($columnlist)"; } if ($self->{use_tablespace} && $self->{tables}{$tbsaved}{idx_tbsp}{$index_name} && !grep(/^$self->{tables}{$tbsaved}{idx_tbsp}{$index_name}$/i, @{$self->{default_tablespaces}})) { $out .= " USING INDEX TABLESPACE $self->{tables}{$tbsaved}{idx_tbsp}{$index_name}"; } if ($deferrable eq "DEFERRABLE") { $out .= " DEFERRABLE"; if ($deferred eq "DEFERRED") { $out .= " INITIALLY DEFERRED"; } } $out .= ";\n"; } } return $out; } =head2 _create_check_constraint This function return SQL code to create the check constraints of a table =cut sub _create_check_constraint { my ($self, $table, $check_constraint, $field_name, @skip_column_check) = @_; my $tbsaved = $table; $table = $self->get_replaced_tbname($table); my $out = ''; # Set the check constraint definition foreach my $k (sort keys %{$check_constraint->{constraint}}) { my $chkconstraint = $check_constraint->{constraint}->{$k}{condition}; my $validate = ''; $validate = ' NOT VALID' if ($check_constraint->{constraint}->{$k}{validate} eq 'NOT VALIDATED'); next if (!$chkconstraint); if ($chkconstraint =~ /^([^\s]+)\s+IS\s+NOT\s+NULL$/i) { my $col = $1; $col =~ s/"//g; if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { next if (uc($col) ne uc($c)); $col = $self->{replaced_cols}{"\L$tbsaved\E"}{$c}; } } $out .= "ALTER TABLE $table ALTER COLUMN " . $self->quote_object_name($col) . " SET NOT NULL;\n"; } else { if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { $chkconstraint =~ s/"$c"/"$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}"/gsi; $chkconstraint =~ s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/gsi; } } if ($self->{plsql_pgsql}) { $chkconstraint = Ora2Pg::PLSQL::convert_plsql_code($self, $chkconstraint); } foreach my $c (@$field_name) { my $ret = $self->quote_object_name($c); $chkconstraint =~ s/\b$c\b/$ret/igs; $chkconstraint =~ s/""/"/igs; } $k = $self->quote_object_name($k); # If the column has been converted as a boolean do not export the constraint my $converted_as_boolean = 0; foreach my $c (@$field_name) { if (grep(/^$c$/i, @skip_column_check) && $chkconstraint =~ /\b$c\b/i) { $converted_as_boolean = 1; } } if (!$converted_as_boolean) { $chkconstraint = Ora2Pg::PLSQL::convert_plsql_code($self, $chkconstraint); $chkconstraint =~ s/,$//; $out .= "ALTER TABLE $table DROP CONSTRAINT $self->{pg_supports_ifexists} $k;\n" if ($self->{drop_if_exists}); $out .= "ALTER TABLE $table ADD CONSTRAINT $k CHECK ($chkconstraint)$validate;\n"; } } } return $out; } =head2 _create_foreign_keys This function return SQL code to create the foreign keys of a table =cut sub _create_foreign_keys { my ($self, $table) = @_; my @out = (); my $tbsaved = $table; $table = $self->get_replaced_tbname($table); # Add constraint definition my @done = (); foreach my $fkname (sort keys %{$self->{tables}{$tbsaved}{foreign_link}}) { next if (grep(/^$fkname$/, @done)); # Extract all attributes if the foreign key definition my $state; foreach my $h (@{$self->{tables}{$tbsaved}{foreign_key}}) { if (lc($h->[0]) eq lc($fkname)) { # @$h : CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE,VALIDATED push(@$state, @$h); last; } } foreach my $desttable (sort keys %{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}}) { push(@done, $fkname); # This is not possible to reference a partitionned table next if ($self->{pg_supports_partition} && exists $self->{partitions_list}{lc($desttable)} && $self->{pg_version} <= 12); # Foreign key constraint on partitionned table do not support # NO VALID when the remote table is not partitionned my $allow_fk_notvalid = 1; $allow_fk_notvalid = 0 if ($self->{pg_supports_partition} && exists $self->{partitions_list}{lc($tbsaved)}); my $str = ''; # Add double quote to column name map { $_ = '"' . $_ . '"' } @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{local}}; map { $_ = '"' . $_ . '"' } @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}{$desttable}}; # Get the name of the foreign table after replacement if any my $subsdesttable = $self->get_replaced_tbname($desttable); # Prefix the table name with the schema name if owner of # remote table is not the same as local one if ($self->{schema} && (lc($state->[6]) ne lc($state->[8]))) { $subsdesttable = $self->quote_object_name($state->[6]) . '.' . $subsdesttable; } my @lfkeys = (); push(@lfkeys, @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{local}}); if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { map { s/"$c"/"$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}"/i } @lfkeys; } } my @rfkeys = (); push(@rfkeys, @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}{$desttable}}); if (exists $self->{replaced_cols}{"\L$desttable\E"} && $self->{replaced_cols}{"\L$desttable\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$desttable\E"}}) { map { s/"$c"/"$self->{replaced_cols}{"\L$desttable\E"}{"\L$c\E"}"/i } @rfkeys; } } for (my $i = 0; $i <= $#lfkeys; $i++) { $lfkeys[$i] = $self->quote_object_name(split(/\s*,\s*/, $lfkeys[$i])); } for (my $i = 0; $i <= $#rfkeys; $i++) { $rfkeys[$i] = $self->quote_object_name(split(/\s*,\s*/, $rfkeys[$i])); } $fkname = $self->quote_object_name($fkname); $str .= "ALTER TABLE $table DROP CONSTRAINT $self->{pg_supports_ifexists} $fkname;\n" if ($self->{drop_if_exists}); my $reftable = $table; $reftable = $self->{partitions_list}{"\L$table\E"}{refrtable} if (exists $self->{partitions_list}{"\L$table\E"}{refrtable}); foreach my $k (keys %{ $self->{tables}{"$reftable"}{column_info} }) { next if (!grep(/^$k$/i, @{$self->{partitions_list}{"\L$reftable\E"}{columns}})); my $f = $self->{tables}{"$reftable"}{column_info}{$k}; # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}; } push(@lfkeys, $self->quote_object_name($fname)); push(@rfkeys, $self->quote_object_name($fname)); } $str .= "ALTER TABLE $table ADD CONSTRAINT $fkname FOREIGN KEY (" . join(',', @lfkeys) . ") REFERENCES $subsdesttable(" . join(',', @rfkeys) . ")"; $str .= " MATCH $state->[2]" if ($state->[2]); if ($state->[3]) { $str .= " ON DELETE $state->[3]"; } else { $str .= " ON DELETE NO ACTION"; } if ($self->{is_mysql}) { $str .= " ON UPDATE $state->[9]" if ($state->[9]); } else { if ( ($self->{fkey_add_update} eq 'ALWAYS') || ( ($self->{fkey_add_update} eq 'DELETE') && ($str =~ /ON DELETE CASCADE/) ) ) { $str .= " ON UPDATE CASCADE"; } } # if DEFER_FKEY is enabled, force constraint to be # deferrable and defer it initially. if (!$self->{is_mysql}) { $str .= (($self->{'defer_fkey'} ) ? ' DEFERRABLE' : " $state->[4]") if ($state->[4]); $state->[5] = 'DEFERRED' if ($state->[5] =~ /^Y/); $state->[5] ||= 'IMMEDIATE'; $str .= " INITIALLY " . ( ($self->{'defer_fkey'} ) ? 'DEFERRED' : $state->[5] ); if ($allow_fk_notvalid && $state->[9] eq 'NOT VALIDATED') { $str .= " NOT VALID"; } } $str .= ";\n"; push(@out, $str); } } return wantarray ? @out : join("\n", @out); } =head2 _drop_foreign_keys This function return SQL code to the foreign keys of a table =cut sub _drop_foreign_keys { my ($self, $table, @foreign_key) = @_; my @out = (); $table = $self->get_replaced_tbname($table); # Add constraint definition my @done = (); foreach my $h (@foreign_key) { next if (grep(/^$h->[0]$/, @done)); push(@done, $h->[0]); my $str = ''; $h->[0] = $self->quote_object_name($h->[0]); $str .= "ALTER TABLE $table DROP CONSTRAINT $self->{pg_supports_ifexists} $h->[0];"; push(@out, $str); } return wantarray ? @out : join("\n", @out); } =head2 _extract_sequence_info This function retrieves the last value returned from the sequences in the Oracle database. The result is a SQL script assigning the new start values to the sequences found in the Oracle database. =cut sub _extract_sequence_info { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_extract_sequence_info($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_extract_sequence_info($self); } else { return Ora2Pg::Oracle::_extract_sequence_info($self); } } =head2 _howto_get_data TABLE This function implements an Oracle-native data extraction. Returns the SQL query to use to retrieve data =cut sub _howto_get_data { my ($self, $table, $name, $type, $src_type, $part_name, $is_subpart) = @_; #### # Overwrite the query if REPLACE_QUERY is defined for this table #### if ($self->{replace_query}{"\L$table\E"}) { $str = $self->{replace_query}{"\L$table\E"}; $self->logit("DEGUG: Query sent to $self->{sgbd_name}: $str\n", 1); return $str; } # Fix a problem when the table need to be prefixed by the schema my $realtable = $table; $realtable =~ s/\"//g; # Do not use double quote with mysql, but backquote if ($self->{is_mysql}) { $realtable =~ s/\`//g; $realtable = "\`$realtable\`"; } elsif ($self->{is_mssql}) { $realtable =~ s/[\[\]]+//g; $realtable = "\[$realtable\]"; if (!$self->{schema} && $self->{export_schema}) { $realtable =~ s/\./\].\[/; } } else { if (!$self->{schema} && $self->{export_schema}) { $realtable =~ s/\./"."/; $realtable = "\"$realtable\""; $reftable = "\"$reftable\""; } else { $realtable = "\"$realtable\""; $reftable = "\"$reftable\""; my $owner = $self->{tables}{$table}{table_info}{owner} || $self->{tables}{$table}{owner} || ''; if ($owner) { $owner =~ s/\"//g; $owner = "\"$owner\""; $realtable = "$owner.$realtable"; $reftable = "$owner.$reftable"; } } } delete $self->{nullable}{$table}; my $alias = 'a'; my $str = "SELECT "; if ($self->{tables}{$table}{table_info}{nested} eq 'YES') { $str = "SELECT /*+ nested_table_get_refs */ "; } if ($self->{is_mssql} && $self->{select_top}) { $str .= "TOP $self->{select_top} "; } my $reftable = $table; my $refcolumn_dst = ''; my $refcolumn_src = ''; my @lfkeys = (); my @rfkeys = (); if ($self->{partition_by_reference} eq 'duplicate' && exists $self->{partitions_list}{"\L$table\E"}{refrtable}) { $reftable = $self->{partitions_list}{"\L$table\E"}{refrtable}; my $fkname = $self->{partitions_list}{"\L$table\E"}{refconstraint}; foreach my $desttable (sort keys %{$self->{tables}{$table}{foreign_link}{$fkname}{remote}}) { next if ($desttable ne $reftable); # Foreign key constraint on partitionned table do not support # NO VALID when the remote table is not partitionned my $allow_fk_notvalid = 1; $allow_fk_notvalid = 0 if ($self->{pg_supports_partition} && exists $self->{partitions_list}{lc($table)}); my $str = ''; # Add double quote to column name map { $_ = '"' . $_ . '"' } @{$self->{tables}{$table}{foreign_link}{$fkname}{local}}; map { $_ = '"' . $_ . '"' } @{$self->{tables}{$table}{foreign_link}{$fkname}{remote}{$desttable}}; # Get the name of the foreign table after replacement if any my $subsdesttable = $self->get_replaced_tbname($desttable); # Prefix the table name with the schema name if owner of # remote table is not the same as local one if ($self->{schema} && (lc($state->[6]) ne lc($state->[8]))) { $subsdesttable = $self->quote_object_name($state->[6]) . '.' . $subsdesttable; } push(@lfkeys, @{$self->{tables}{$table}{foreign_link}{$fkname}{local}}); if (exists $self->{replaced_cols}{"\L$table\E"} && $self->{replaced_cols}{"\L$table\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$table\E"}}) { map { s/"$c"/"$self->{replaced_cols}{"\L$table\E"}{"\L$c\E"}"/i } @lfkeys; } } push(@rfkeys, @{$self->{tables}{$table}{foreign_link}{$fkname}{remote}{$desttable}}); if (exists $self->{replaced_cols}{"\L$desttable\E"} && $self->{replaced_cols}{"\L$desttable\E"}) { foreach my $c (keys %{$self->{replaced_cols}{"\L$desttable\E"}}) { map { s/"$c"/"$self->{replaced_cols}{"\L$desttable\E"}{"\L$c\E"}"/i } @rfkeys; } } } $refcolumn_src = $self->{partitions_list}{"\L$table\E"}{refcolumn}; foreach my $k (keys %{ $self->{tables}{"$reftable"}{column_info} }) { next if (!grep(/^$k$/i, @{$self->{partitions_list}{"\L$reftable\E"}{columns}})); my $f = $self->{tables}{"$reftable"}{column_info}{$k}; # Change column names my $fname = $f->[0]; if (exists $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}) { $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"} . "...\n", 1); $fname = $self->{replaced_cols}{"\L$reftable\E"}{"\L$fname\E"}; } $refcolumn_dst = $fname; } } my $extraStr = ""; # Lookup through columns information if ($#{$name} < 0) { # There a problem whe can't find any column in this table return ''; } else { for my $k (0 .. $#{$name}) { my $realcolname = $name->[$k]; my $spatial_srid = ''; $self->{nullable}{$table}{$k} = $self->{colinfo}->{$table}{$realcolname}{nullable}; if ($self->{is_mysql}) { if ($name->[$k] !~ /\`/) { $name->[$k] = '`' . $name->[$k] . '`'; } } elsif ($self->{is_mssql}) { #if ($name->[$k] !~ /\[/) { $name->[$k] = '[' . $name->[$k] . ']'; #} } else { if ($name->[$k] !~ /"/) { $name->[$k] = '"' . $name->[$k] . '"'; } } # If there is any transformation to apply replace the column name with the clause if (exists $self->{transform_value}{lc($table)} && exists $self->{transform_value}{lc($table)}{lc($realcolname)}) { $str .= $self->{transform_value}{lc($table)}{lc($realcolname)} . ","; } # Apply some default transformation following the data type elsif ( ( $src_type->[$k] =~ /^char/i) && ($type->[$k] =~ /(varchar|text)/i)) { my $colnm = $name->[$k]; $colnm =~ s/^[^\.]+\.//; $str .= "trim($self->{trim_type} '$self->{trim_char}' FROM $alias.$name->[$k]) AS $colnm,"; } elsif ($self->{is_mysql} && $src_type->[$k] =~ /bit/i) { $str .= "BIN($alias.$name->[$k]),"; } # If dest type is bytea the content of the file is exported as bytea elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /bytea/i) ) { $self->{bfile_found} = 'bytea'; $str .= "ora2pg_get_bfile($alias.$name->[$k]),"; } # If dest type is efile the content of the file is exported to use the efile extension elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /efile/i) ) { $self->{bfile_found} = 'efile'; $str .= "ora2pg_get_efile($alias.$name->[$k]),"; } # Only extract path to the bfile if dest type is text. elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /text/i) ) { $self->{bfile_found} = 'text'; $str .= "ora2pg_get_bfilename($alias.$name->[$k]),"; } elsif ( $src_type->[$k] =~ /xmltype/i) { if ($self->{xml_pretty}) { $str .= "($alias.$name->[$k]).getStringVal(),"; } else { $str .= "($alias.$name->[$k]).getClobVal(),"; } } # ArcGis Geometries elsif ( !$self->{is_mysql} && $src_type->[$k] =~ /^(ST_|STGEOM_)/i) { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN $self->{st_asbinary_function}($alias.$name->[$k]) ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN $self->{st_astext_function}($alias.$name->[$k]) ELSE NULL END,"; } } # Oracle geometries elsif ( !$self->{is_mysql} && $src_type->[$k] =~ /SDO_GEOMETRY/i) { # Set SQL query to get the SRID of the column if ($self->{convert_srid} > 1) { $spatial_srid = $self->{convert_srid}; } else { $spatial_srid = $self->{colinfo}->{$table}{$realcolname}{spatial_srid}; } # With INSERT statement we always use WKT if ($self->{type} eq 'INSERT') { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN SDO_UTIL.TO_WKBGEOMETRY($alias.$name->[$k]) ELSE NULL END,"; } elsif ($self->{geometry_extract_type} eq 'INTERNAL') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN $alias.$name->[$k] ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN ST_GeomFromText(SDO_UTIL.TO_WKTGEOMETRY($alias.$name->[$k]), '$spatial_srid') ELSE NULL END,"; } } else { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN SDO_UTIL.TO_WKBGEOMETRY($alias.$name->[$k]) ELSE NULL END,"; } elsif ($self->{geometry_extract_type} eq 'INTERNAL') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN $alias.$name->[$k] ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN SDO_UTIL.TO_WKTGEOMETRY($alias.$name->[$k]) ELSE NULL END,"; } } } # SQL Server geometry elsif ( $self->{is_mssql} && $src_type->[$k] =~ /^GEOM(ETRY|GRAPHY)/i) { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=', $alias.$name->[$k].STSrid,';', $alias.$name->[$k].STAsText()) ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=', $alias.$name->[$k].STSrid,';', $alias.$name->[$k].STAsText()) ELSE NULL END,"; } } # MySQL geometry elsif ( $self->{is_mysql} && $src_type->[$k] =~ /geometry/i && $self->{type} ne 'TEST_DATA') { if ($self->{db_version} < '5.7.6') { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=',SRID($alias.$name->[$k]),';', AsBinary($alias.$name->[$k])) ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=',SRID($alias.$name->[$k]),';', AsText($alias.$name->[$k])) ELSE NULL END,"; } } else { if ($self->{geometry_extract_type} eq 'WKB') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=',$self->{st_srid_function}($alias.$name->[$k]),';', $self->{st_asbinary_function}($alias.$name->[$k])) ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN CONCAT('SRID=',$self->{st_srid_function}($alias.$name->[$k]),';', $self->{st_astext_function}($alias.$name->[$k])) ELSE NULL END,"; } } } # For data testing we retrieve geometry using ST_AsText/AsText elsif ( $self->{is_mysql} && $src_type->[$k] =~ /geometry/i && $self->{type} eq 'TEST_DATA') { if ($self->{db_version} < '5.7.6') { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN AsText($alias.$name->[$k]) ELSE NULL END,"; } else { $str .= "CASE WHEN $alias.$name->[$k] IS NOT NULL THEN $self->{st_astext_function}($alias.$name->[$k]) ELSE NULL END,"; } } elsif ( !$self->{is_mysql} && (($src_type->[$k] =~ /clob/i) || ($src_type->[$k] =~ /blob/i)) ) { if (!$self->{enable_blob_export} && $src_type->[$k] =~ /blob/i) { # user don't want to export blob next; } if (!$self->{enable_clob_export} && $src_type->[$k] =~ /clob/i) { # user don't want to export clob next; } if ($self->{empty_lob_null}) { $str .= "CASE WHEN "; if (!$self->{has_dbms_log_execute_privilege}) { if ($src_type->[$k] =~ /blob/i) { $str .= "LENGTHB"; } else { $str .= "LENGTH"; } } else { $str .= "dbms_lob.getlength"; } $str .= "($alias.$name->[$k]) = 0 THEN NULL ELSE $alias.$name->[$k] END,"; } else { $str .= "$alias.$name->[$k],"; } } else { $str .= "$alias.$name->[$k],"; } push(@{$self->{spatial_srid}{$table}}, $spatial_srid); if ( ($type->[$k] =~ /bytea/i && $self->{enable_blob_export}) || ($self->{clob_as_blob} && $src_type->[$k] =~ /CLOB/i) ) { if ($self->{data_limit} >= 1000) { $self->{local_data_limit}{$table} = int($self->{data_limit}/10); while ($self->{local_data_limit}{$table} > 1000) { $self->{local_data_limit}{$table} = int($self->{local_data_limit}{$table}/10); } } else { $self->{local_data_limit}{$table} = $self->{data_limit}; } $self->{local_data_limit}{$table} = $self->{blob_limit} if ($self->{blob_limit}); } } $str =~ s/,$//; } # If we have a BFILE that might be exported as text we need to create a function my $bfile_function = ''; if ($self->{bfile_found} eq 'text') { $self->logit("Creating function ora2pg_get_bfilename( p_bfile IN BFILE ) to retrieve path from BFILE.\n", 1); $bfile_function = qq{ CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); l_path VARCHAR2(4000); BEGIN IF p_bfile IS NULL THEN RETURN NULL; ELSE dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); SELECT DIRECTORY_PATH INTO l_path FROM $self->{prefix}_DIRECTORIES WHERE DIRECTORY_NAME = l_dir; l_dir := rtrim(l_path,'/'); RETURN l_dir || '/' || replace(l_fname, '\\', '/'); END IF; END; }; } # If we have a BFILE that might be exported as efile we need to create a function elsif ($self->{bfile_found} eq 'efile') { $self->logit("Creating function ora2pg_get_efile( p_bfile IN BFILE ) to retrieve EFILE from BFILE.\n", 1); my $quote = ''; $quote = "''" if ($self->{type} eq 'INSERT'); $bfile_function = qq{ CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) RETURN VARCHAR2 AS l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); BEGIN IF p_bfile IS NULL THEN RETURN NULL; ELSE dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); RETURN '($quote' || l_dir || '$quote,$quote' || replace(l_fname, '\\', '/') || '$quote)'; END IF; END; }; } # If we have a BFILE that might be exported as bytea we need to create a # function that exports the bfile as a binary BLOB, a HEX encoded string elsif ($self->{bfile_found} eq 'bytea') { $self->logit("Creating function ora2pg_get_bfile( p_bfile IN BFILE ) to retrieve BFILE content as BLOB.\n", 1); $bfile_function = qq{ CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN BLOB AS filecontent BLOB := NULL; src_file BFILE := NULL; l_step PLS_INTEGER := 12000; l_dir VARCHAR2(4000); l_fname VARCHAR2(4000); offset NUMBER := 1; BEGIN IF p_bfile IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); src_file := BFILENAME( l_dir, l_fname ); IF src_file IS NULL THEN RETURN NULL; END IF; DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); DBMS_LOB.CREATETEMPORARY(filecontent, true); DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); DBMS_LOB.FILECLOSE(src_file); RETURN filecontent; END; }; } if ($bfile_function) { my $local_dbh = $self->_db_connection(); my $sth2 = $local_dbh->do($bfile_function); $local_dbh->disconnect() if ($local_dbh); } # Fix empty column list with nested table if (!$self->{is_mysql}) { $str =~ s/ ""$/ \*/; } else { $str =~ s/ ``$/ \*/; } if ($part_name) { if ($is_subpart && !$self->{is_mysql}) { $realtable .= " SUBPARTITION(" . $self->quote_object_name($part_name) . ")"; } else { $realtable .= " PARTITION(" . $self->quote_object_name($part_name) . ")"; } } # Force parallelism on Oracle side if ($self->{default_parallelism_degree} > 1 && $self->{type} ne 'TEST_DATA') { # Only if the number of rows is upper than PARALLEL_MIN_ROWS $self->{tables}{$table}{table_info}{num_rows} ||= 0; if ($self->{tables}{$table}{table_info}{num_rows} > $self->{parallel_min_rows}) { $str =~ s#^SELECT #SELECT /*+ FULL(a) PARALLEL(a, $self->{default_parallelism_degree}) */ #; } } $str .= " FROM $realtable"; if ($self->{start_scn} =~ /^\d+$/) { $str .= " AS OF SCN $self->{start_scn}"; } elsif ($self->{start_scn}) { $str .= " AS OF TIMESTAMP $self->{start_scn}"; } elsif (exists $self->{current_oracle_scn}{$table}) { $str .= " AS OF SCN $self->{current_oracle_scn}{$table}"; } $str .= " $alias"; if ($refcolumn_dst) { $str .= " JOIN $reftable"; if ($self->{start_scn} =~ /^\d+$/) { $str .= " AS OF SCN $self->{start_scn}"; } elsif ($self->{start_scn}) { $str .= " AS OF TIMESTAMP $self->{start_scn}"; } elsif (exists $self->{current_oracle_scn}{$table}) { $str .= " AS OF SCN $self->{current_oracle_scn}{$table}"; } # The partition by reference column, doesn't exist in the Oracle child table. Use the origin. $str =~ s/,$alias\."$refcolumn_dst"/,reftb."$refcolumn_dst"/i; $str .= " reftb ON ("; map { s/^(.*)$/$alias\.$1/; } @lfkeys; map { s/^(.*)$/reftb\.$1/; } @rfkeys; for (my $k = 0; $k <= $#lfkeys; $k++) { $lfkeys[$k] =~ s/["]+/"/g; $rfkeys[$k] =~ s/["]+/"/g; $str .= "$lfkeys[$k] = $rfkeys[$k] AND"; } $str =~ s/ AND$/)/; } if (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}) { ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; if (!$self->{is_mysql} || ($self->{where}{"\L$table\E"} !~ /\s+LIMIT\s+\d/)) { $str .= '(' . $self->{where}{"\L$table\E"} . ')'; } else { $str .= $self->{where}{"\L$table\E"}; } $self->logit("\tApplying WHERE clause on table: " . $self->{where}{"\L$table\E"} . "\n", 1); } elsif ($self->{global_where}) { ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; if (!$self->{is_mysql} || ($self->{global_where} !~ /\s+LIMIT\s+\d/)) { $str .= '(' . $self->{global_where} . ')'; } else { $str .= $self->{global_where}; } $self->logit("\tApplying WHERE global clause: " . $self->{global_where} . "\n", 1); } # Automatically set the column on which query will be splitted # to the first column with a unique key and of type NUMBER. if ($self->{oracle_copies} > 1 && $self->{type} ne 'TEST_DATA') { if (!exists $self->{defined_pk}{"\L$table\E"}) { foreach my $consname (keys %{$self->{tables}{$table}{unique_key}}) { my $constype = $self->{tables}{$table}{unique_key}->{$consname}{type}; if (($constype eq 'P') || ($constype eq 'U')) { foreach my $c (@{$self->{tables}{$table}{unique_key}->{$consname}{columns}}) { for my $k (0 .. $#{$name}) { my $realcolname = $name->[$k]->[0]; $realcolname =~ s/"//g; if ($c eq $realcolname) { if ($src_type->[$k] =~ /^number\(.*,.*\)/i) { $self->{defined_pk}{"\L$table\E"} = "ROUND($c)"; last; } elsif ($src_type->[$k] =~ /^number/i) { $self->{defined_pk}{"\L$table\E"} = $c; last; } } } last if (exists $self->{defined_pk}{"\L$table\E"}); } } last if (exists $self->{defined_pk}{"\L$table\E"}); } } if ($self->{defined_pk}{"\L$table\E"}) { my $colpk = $self->{defined_pk}{"\L$table\E"}; if ($self->{preserve_case}) { $colpk = '"' . $colpk . '"'; } if ($str =~ / WHERE /) { $str .= " AND"; } else { $str .= " WHERE"; } if ($self->{is_mssql}) { $str .= " ABS($colpk % $self->{oracle_copies}) = ?"; } else { $str .= " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; } } } $self->logit("DEGUG: Query sent to $self->{sgbd_name}: $str\n", 1); return $str; } =head2 _howto_get_fdw_data TABLE This function implements an Oracle data extraction through oracle_fdw. Returns the SQL query to use to retrieve data =cut sub _howto_get_fdw_data { my ($self, $table, $name, $type, $src_type, $part_name, $is_subpart) = @_; #### # Overwrite the query if REPLACE_QUERY is defined for this table #### if ($self->{replace_query}{"\L$table\E"}) { $str = $self->{replace_query}{"\L$table\E"}; $self->logit("DEGUG: Query sent to $self->{sgbd_name}: $str\n", 1); return $str; } # Fix a problem when the table need to be prefixed by the schema my $realtable = $table; $realtable =~ s/\"//g; # Do not use double quote with mysql, but backquote if (!$self->{is_mysql}) { if (!$self->{schema} && $self->{export_schema}) { $realtable =~ s/\./"."/; $realtable = "\"$realtable\""; } else { $realtable = "\"$realtable\""; my $owner = $self->{tables}{$table}{table_info}{owner} || $self->{tables}{$table}{owner} || ''; if ($owner) { $owner =~ s/\"//g; $owner = "\"$owner\""; $realtable = "$owner.$realtable"; } } } else { $realtable = "\`$realtable\`"; } delete $self->{nullable}{$table}; my $alias = 'a'; my $str = "SELECT "; my $extraStr = ""; # Lookup through columns information if ($#{$name} < 0) { # There a problem whe can't find any column in this table return ''; } else { for my $k (0 .. $#{$name}) { my $realcolname = $name->[$k]->[0]; my $spatial_srid = ''; $self->{nullable}{$table}{$k} = $self->{colinfo}->{$table}{$realcolname}{nullable}; if ($name->[$k]->[0] !~ /"/) { # Do not use double quote with mysql if (!$self->{is_mysql}) { $name->[$k]->[0] = '"' . $name->[$k]->[0] . '"'; } else { $name->[$k]->[0] = '`' . $name->[$k]->[0] . '`'; } } # If dest type is bytea the content of the file is exported as bytea if ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /bytea/i) ) { $self->{bfile_found} = 'bytea'; } # If dest type is efile the content of the file is exported to use the efile extension elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /efile/i) ) { $self->{bfile_found} = 'efile'; $self->logit("FATAL: with oracle_fdw data export, BFILE can only be converted to bytea\n", 0, 1); } # Only extract path to the bfile if dest type is text. elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /text/i) ) { $self->{bfile_found} = 'text'; $self->logit("FATAL: with oracle_fdw data export, BFILE can only be converted to bytea\n", 0, 1); } $str .= "$name->[$k]->[0],"; push(@{$self->{spatial_srid}{$table}}, $spatial_srid); # Wit oracle_fdw export we migrate data in stream not in chunk $self->{data_limit} = 0; $self->{local_data_limit}{$table} = 0; $self->{blob_limit} = 0; } $str =~ s/,$//; } $str .= " FROM $realtable $alias"; if (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}) { ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; if (!$self->{is_mysql} || ($self->{where}{"\L$table\E"} !~ /\s+LIMIT\s+\d/)) { $str .= '(' . $self->{where}{"\L$table\E"} . ')'; } else { $str .= $self->{where}{"\L$table\E"}; } $self->logit("\tApplying WHERE clause on table: " . $self->{where}{"\L$table\E"} . "\n", 1); } elsif ($self->{global_where}) { ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; if (!$self->{is_mysql} || ($self->{global_where} !~ /\s+LIMIT\s+\d/)) { $str .= '(' . $self->{global_where} . ')'; } else { $str .= $self->{global_where}; } $self->logit("\tApplying WHERE global clause: " . $self->{global_where} . "\n", 1); } # Automatically set the column on which query will be splitted # to the first column with a unique key and of type NUMBER. if ($self->{oracle_copies} > 1) { if (!exists $self->{defined_pk}{"\L$table\E"}) { foreach my $consname (keys %{$self->{tables}{$table}{unique_key}}) { my $constype = $self->{tables}{$table}{unique_key}->{$consname}{type}; if (($constype eq 'P') || ($constype eq 'U')) { foreach my $c (@{$self->{tables}{$table}{unique_key}->{$consname}{columns}}) { for my $k (0 .. $#{$name}) { my $realcolname = $name->[$k]->[0]; $realcolname =~ s/"//g; if ($c eq $realcolname) { if ($src_type->[$k] =~ /^number\(.*,.*\)/i) { $self->{defined_pk}{"\L$table\E"} = "ROUND($c)"; last; } elsif ($src_type->[$k] =~ /^number/i) { $self->{defined_pk}{"\L$table\E"} = $c; last; } } } last if (exists $self->{defined_pk}{"\L$table\E"}); } } last if (exists $self->{defined_pk}{"\L$table\E"}); } } if ($self->{defined_pk}{"\L$table\E"}) { my $colpk = $self->{defined_pk}{"\L$table\E"}; if ($self->{preserve_case}) { $colpk = '"' . $colpk . '"'; } if ($str =~ / WHERE /) { $str .= " AND"; } else { $str .= " WHERE"; } if ($self->{is_mssql}) { $str .= " ABS($colpk % $self->{oracle_copies}) = ?"; } else { $str .= " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; } } } $self->logit("DEGUG: Query sent to $self->{sgbd_name}: $str\n", 1); return $str; } =head2 _sql_type INTERNAL_TYPE LENGTH PRECISION SCALE This function returns the PostgreSQL data type corresponding to the Oracle data type. =cut sub _sql_type { my ($self, $type, $len, $precision, $scale, $default, $no_blob_to_oid) = @_; $type = uc($type); # Force uppercase if ($self->{is_mysql}) { return Ora2Pg::MySQL::_sql_type($self, $type, $len, $precision, $scale); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_sql_type($self, $type, $len, $precision, $scale); } else { return Ora2Pg::Oracle::_sql_type($self, $type, $len, $precision, $scale); } } =head2 _column_info TABLE OWNER This function implements an Oracle-native column information. Returns a list of array references containing the following information elements for each column the specified table [( column name, column type, column length, nullable column, default value ... )] =cut sub _column_info { my ($self, $table, $owner, $objtype, @expanded_views) = @_; $objtype ||= 'TABLE'; $self->logit("Collecting column information for \L$objtype\E $table...\n", 1); if ($self->{is_mysql}) { return Ora2Pg::MySQL::_column_info($self,$table,$owner,$objtype,0,@expanded_views); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_column_info($self,$table,$owner,$objtype,0,@expanded_views); } else { return Ora2Pg::Oracle::_column_info($self,$table,$owner,$objtype,0,@expanded_views); } } sub _column_attributes { my ($self, $table, $owner, $objtype) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_column_attributes($self,$table,$owner,'TABLE'); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_column_attributes($self,$table,$owner,'TABLE'); } else { return Ora2Pg::Oracle::_column_attributes($self,$table,$owner,'TABLE'); } } sub _encrypted_columns { my ($self, $table, $owner) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_encrypted_columns($self,$table,$owner); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_encrypted_columns($self,$table,$owner); } else { return Ora2Pg::Oracle::_encrypted_columns($self,$table,$owner); } } =head2 _unique_key TABLE OWNER This function implements an Oracle-native unique (including primary) key column information. Returns a hash of hashes in the following form: ( owner => table => constraintname => (type => 'PRIMARY', columns => ('a', 'b', 'c')), owner => table => constraintname => (type => 'UNIQUE', columns => ('b', 'c', 'd')), etc. ) =cut sub _unique_key { my ($self, $table, $owner, $type) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_unique_key($self,$table,$owner, $type); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_unique_key($self,$table,$owner, $type); } else { return Ora2Pg::Oracle::_unique_key($self,$table,$owner, $type); } } =head2 _check_constraint TABLE OWNER This function implements an Oracle-native check constraint information. Returns a hash of lists of all column names defined as check constraints for the specified table and constraint name. =cut sub _check_constraint { my ($self, $table, $owner) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_check_constraint($self, $table, $owner); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_check_constraint($self, $table, $owner); } else { return Ora2Pg::Oracle::_check_constraint($self, $table, $owner); } } =head2 _foreign_key TABLE OWNER This function implements an Oracle-native foreign key reference information. Returns a list of hash of hash of array references. Ouf! Nothing very difficult. The first hash is composed of all foreign key names. The second hash has just two keys known as 'local' and 'remote' corresponding to the local table where the foreign key is defined and the remote table referenced by the key. The foreign key name is composed as follows: 'local_table_name->remote_table_name' Foreign key data consists in two arrays representing at the same index for the local field and the remote field where the first one refers to the second one. Just like this: @{$link{$fkey_name}{local}} = @local_columns; @{$link{$fkey_name}{remote}} = @remote_columns; =cut sub _foreign_key { my ($self, $table, $owner) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_foreign_key($self,$table,$owner); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_foreign_key($self,$table,$owner); } else { return Ora2Pg::Oracle::_foreign_key($self,$table,$owner); } } =head2 _get_privilege This function implements an Oracle-native object priviledge information. Returns a hash of all priviledge. =cut sub _get_privilege { my($self) = @_; # If the user is given as not DBA, do not look at tablespace if ($self->{user_grants}) { $self->logit("WARNING: Exporting privilege as non DBA user is not allowed, see USER_GRANT\n", 0); return; } if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_privilege($self); } else { return Ora2Pg::Oracle::_get_privilege($self); } } =head2 _get_security_definer This function implements an Oracle-native functions security definer / current_user information. Returns a hash of all object_type/function/security. =cut sub _get_security_definer { my ($self, $type) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_security_definer($self, $type); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_security_definer($self, $type); } else { return Ora2Pg::Oracle::_get_security_definer($self, $type); } } =head2 _get_indexes TABLE OWNER This function implements an Oracle-native indexes information. Returns a hash of an array containing all unique indexes and a hash of array of all indexe names which are not primary keys for the specified table. =cut sub _get_indexes { my ($self, $table, $owner, $generated_indexes) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_indexes($self,$table,$owner, $generated_indexes); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_indexes($self,$table,$owner, $generated_indexes); } else { return Ora2Pg::Oracle::_get_indexes($self,$table,$owner, $generated_indexes); } } =head2 _get_sequences This function implements an Oracle-native sequences information. Returns a hash of an array of sequence names with MIN_VALUE, MAX_VALUE, INCREMENT and LAST_NUMBER for the specified table. =cut sub _get_sequences { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_sequences($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_sequences($self); } else { return Ora2Pg::Oracle::_get_sequences($self); } } =head2 _get_identities This function retrieve information about IDENTITY columns that must be exported as PostgreSQL serial. =cut sub _get_identities { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_identities($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_identities($self); } else { return Ora2Pg::Oracle::_get_identities($self); } } =head2 _get_external_tables This function implements an Oracle-native external tables information. Returns a hash of external tables names with the file they are based on. =cut sub _get_external_tables { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_external_tables($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_external_tables($self); } else { return Ora2Pg::Oracle::_get_external_tables($self); } } =head2 _get_directory This function implements an Oracle-native directory information. Returns a hash of directory names with the path they are based on. =cut sub _get_directory { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_directory($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_directory($self); } else { return Ora2Pg::Oracle::_get_directory($self); } } =head2 _get_dblink This function implements an Oracle-native database link information. Returns a hash of dblink names with the connection they are based on. =cut sub _get_dblink { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_dblink($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_dblink($self); } else { return Ora2Pg::Oracle::_get_dblink($self); } } =head2 _get_job This function implements an Oracle-native job information. Reads together from view [ALL|DBA]_JOBS and from view [ALL|DBA]_SCHEDULER_JOBS. Returns a hash of job number with the connection they are based on. =cut sub _get_job { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_job($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_job($self); } else { return Ora2Pg::Oracle::_get_job($self); } } =head2 _get_views This function implements an Oracle-native views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_views { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_views($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_views($self); } else { return Ora2Pg::Oracle::_get_views($self); } } =head2 _get_materialized_views This function implements an Oracle-native materialized views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_materialized_views { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_materialized_views($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_materialized_views($self); } else { return Ora2Pg::Oracle::_get_materialized_views($self); } } sub _get_materialized_view_names { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_materialized_view_names($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_materialized_view_names($self); } else { return Ora2Pg::Oracle::_get_materialized_view_names($self); } } =head2 _get_triggers This function implements an Oracle-native triggers information. Returns an array of refarray of all triggers information. =cut sub _get_triggers { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_triggers($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_triggers($self); } else { return Ora2Pg::Oracle::_get_triggers($self); } } sub _list_triggers { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_list_triggers($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_list_triggers($self); } else { return Ora2Pg::Oracle::_list_triggers($self); } } =head2 _get_plsql_metadata This function retrieve all metadata on Oracle store procedure. Returns a hash of all function names with their metadata information (arguments, return type, etc.). =cut sub _get_plsql_metadata { my $self = shift; my $owner = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_plsql_metadata($self, $owner); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_plsql_metadata($self, $owner); } else { return Ora2Pg::Oracle::_get_plsql_metadata($self, $owner); } } =head2 _get_package_function_list This function retrieve all function and procedure defined on Oracle store procedure PACKAGE. Returns a hash of all package function names =cut sub _get_package_function_list { my $self = shift; my $owner = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_package_function_list($self, $owner); } else { return Ora2Pg::Oracle::_get_package_function_list($self, $owner); } } =head2 _get_functions This function implements an Oracle-native functions information. Returns a hash of all function names with their PLSQL code. =cut sub _get_functions { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_functions($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_functions($self); } else { return Ora2Pg::Oracle::_get_functions($self); } } =head2 _get_procedures This procedure implements an Oracle-native procedures information. Returns a hash of all procedure names with their PLSQL code. =cut sub _get_procedures { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_procedures($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_procedures($self); } else { return Ora2Pg::Oracle::_get_procedures($self); } } =head2 _get_packages This function implements an Oracle-native packages information. Returns a hash of all package names with their PLSQL code. =cut sub _get_packages { my ($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_packages($self); } else { return Ora2Pg::Oracle::_get_packages($self); } } =head2 _get_types This function implements an Oracle custom types information. Returns a hash of all type names with their code. =cut sub _get_types { my ($self, $name) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_types($self, $name); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_types($self, $name); } else { return Ora2Pg::Oracle::_get_types($self, $name); } } =head2 _count_source_rows This function retrieves real rows count from a table. =cut sub _count_source_rows { my ($self, $dbhsrc, $t) = @_; $self->logit("DEBUG: pid $$ looking for real row count for source table $t...\n", 1); my $tbname = $t; if ($self->{is_mysql}) { $tbname = "`$t`"; } elsif ($self->{is_mssql}) { $tbname = "[$t]"; $tbname =~ s/\./\].\[/; } else { $tbname = qq{"$t"}; $tbname = qq{"$self->{schema}"} . '.' . qq{"$t"} if ($self->{schema} && $t !~ /\./); } my $sql = "SELECT COUNT(*) FROM $tbname"; my $sth = $dbhsrc->prepare( $sql ) or $self->logit("FATAL: " . $dbhsrc->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbhsrc->errstr . "\n", 0, 1); my $size = $sth->fetch(); $sth->finish(); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $fh = new IO::File; $fh->open(">>${dirprefix}ora2pg_count_rows") or $self->logit("FATAL: can't write to ${dirprefix}ora2pg_count_rows, $!\n", 0, 1); flock($fh, 2) || die "FATAL: can't lock file ${dirprefix}ora2pg_count_rows\n"; $fh->print("$tbname:$size->[0]\n"); $fh->close; } =head2 _table_info This function retrieves all Oracle-native tables information. Returns a handle to a DB query statement. =cut sub _table_info { my $self = shift; my $do_real_row_count = shift; my %tables_infos = (); if ($self->{is_mysql}) { %tables_infos = Ora2Pg::MySQL::_table_info($self); } elsif ($self->{is_mssql}) { %tables_infos = Ora2Pg::MSSQL::_table_info($self); } else { %tables_infos = Ora2Pg::Oracle::_table_info($self); } # Collect real row count for each table if ($do_real_row_count) { my $t1 = Benchmark->new; my $parallel_tables_count = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); unlink("${dirprefix}ora2pg_count_rows"); foreach my $t (sort keys %tables_infos) { if ($self->{parallel_tables} > 1) { spawn sub { my $dbhsrc = $self->{dbh}->clone(); $self->_count_source_rows($dbhsrc, $t); $dbhsrc->disconnect(); }; $parallel_tables_count++; # Wait for oracle connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } else { $self->_count_source_rows($self->{dbh}, $t); } } # Wait for all child die if ($self->{parallel_tables} > 1) { while (scalar keys %RUNNING_PIDS > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { delete $RUNNING_PIDS{$kid}; } usleep(50000); } } my $fh = new IO::File; $fh->open("${dirprefix}ora2pg_count_rows") or $self->logit("FATAL: can't read file ${dirprefix}ora2pg_count_rows, $!\n", 0, 1); my @ret = <$fh>; $fh->close; unlink("${dirprefix}ora2pg_count_rows"); foreach my $s (@ret) { my ($tb, $cnt) = split(':', $s); chomp $cnt; $tb =~ s/"//g; my ($ora_owner, $ora_table) = split('\.', $tb); if ($tables_infos{$ora_table}{owner} eq $ora_owner) { $tables_infos{$ora_table}{num_rows} = $cnt || 0; } } my $t2 = Benchmark->new; $td = timediff($t2, $t1); $self->logit("Collecting tables real row count took: " . timestr($td) . "\n", 1); } return %tables_infos; } =head2 _global_temp_table_info This function retrieves all Oracle-native global temporary tables information. Returns a handle to a DB query statement. =cut sub _global_temp_table_info { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_global_temp_table_info($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_global_temp_table_info($self); } else { return Ora2Pg::Oracle::_global_temp_table_info($self); } } =head2 _queries This function is used to retrieve all Oracle queries from DBA_AUDIT_TRAIL Sets the main hash $self->{queries}. =cut sub _queries { my ($self) = @_; $self->logit("Retrieving audit queries information...\n", 1); %{$self->{queries}} = $self->_get_audit_queries(); } =head2 _get_audit_queries This function extract SQL queries from dba_audit_trail Returns a hash of queries. =cut sub _get_audit_queries { my($self) = @_; return if (!$self->{audit_user}); # If the user is given as not DBA, do not look at tablespace if ($self->{user_grants}) { $self->logit("WARNING: Exporting audited queries as non DBA user is not allowed, see USER_GRANT\n", 0); return; } if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_audit_queries($self); } else { return Ora2Pg::Oracle::_get_audit_queries($self); } } =head2 _get_tablespaces This function implements an Oracle-native tablespaces information. Returns a hash of an array of tablespace names with their system file path. =cut sub _get_tablespaces { my($self) = @_; # If the user is given as not DBA, do not look at tablespace if ($self->{user_grants}) { $self->logit("WARNING: Exporting tablespace as non DBA user is not allowed, see USER_GRANT\n", 0); return; } if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_tablespaces($self); } else { return Ora2Pg::Oracle::_get_tablespaces($self); } } sub _list_tablespaces { my($self) = @_; # If the user is given as not DBA, do not look at tablespace if ($self->{user_grants}) { return; } if ($self->{is_mysql}) { return Ora2Pg::MySQL::_list_tablespaces($self); } else { return Ora2Pg::Oracle::_list_tablespaces($self); } } =head2 _get_partitions This function implements an Oracle-native partitions information. Return two hash ref with partition details and partition default. =cut sub _get_partitions { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_partitions($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_partitions($self); } else { return Ora2Pg::Oracle::_get_partitions($self); } } =head2 _get_subpartitions This function implements an Oracle-native subpartitions information. Return two hash ref with partition details and partition default. =cut sub _get_subpartitions { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_subpartitions($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_subpartitions($self); } else { return Ora2Pg::Oracle::_get_subpartitions($self); } } =head2 _synonyms This function is used to retrieve all synonyms information. Sets the main hash of the synonyms definition $self->{synonyms}. Keys are the names of all synonyms retrieved from the current database. The synonyms hash is construct as follows: $hash{SYNONYM_NAME}{owner} = Owner of the synonym $hash{SYNONYM_NAME}{table_owner} = Owner of the object referenced by the synonym. $hash{SYNONYM_NAME}{table_name} = Name of the object referenced by the synonym. $hash{SYNONYM_NAME}{dblink} = Name of the database link referenced, if any =cut sub _synonyms { my ($self) = @_; # Get all synonyms information $self->logit("Retrieving synonyms information...\n", 1); %{$self->{synonyms}} = $self->_get_synonyms(); } =head2 _get_synonyms This function implements an Oracle-native synonym information. =cut sub _get_synonyms { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_synonyms($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_synonyms($self); } else { return Ora2Pg::Oracle::_get_synonyms($self); } } =head2 _get_partitions_list This function implements an Oracle-native partitions information. Return a hash of the partition table_name => type =cut sub _get_partitions_list { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_partitions_list($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_partitions_list($self); } else { return Ora2Pg::Oracle::_get_partitions_list($self); } } =head2 _get_partitioned_table Return a hash of the partitioned table list with the number of partition. =cut sub _get_partitioned_table { my ($self, %subpart) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_partitioned_table($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_partitioned_table($self); } else { return Ora2Pg::Oracle::_get_partitioned_table($self); } } =head2 _get_subpartitioned_table Return a hash of the partitioned table list with the number of partition. =cut sub _get_subpartitioned_table { my($self) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_subpartitioned_table($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_subpartitioned_table($self); } else { return Ora2Pg::Oracle::_get_subpartitioned_table($self); } } sub _get_custom_types { my ($self, $str, $parent) = @_; # Copy the type translation hash my %all_types = undef; if ($self->{is_mysql}) { %all_types = %Ora2Pg::MySQL::SQL_TYPE; } elsif ($self->{is_mssql}) { %all_types = %Ora2Pg::MSSQL::SQL_TYPE; } else { %all_types = %Ora2Pg::Oracle::SQL_TYPE; } # replace type double precision by single word double $all_types{'DOUBLE'} = $all_types{'DOUBLE PRECISION'}; delete $all_types{'DOUBLE PRECISION'}; # Remove any parenthesis after a type foreach my $t (keys %all_types) { $str =~ s/$t\s*\([^\)]+\)/$t/igs; } $str =~ s/^[^\(]+\(//s; $str =~ s/\s*\)\s*;$//s; $str =~ s/\/\*(.*?)\*\///gs; $str =~ s/\s*--[^\r\n]+//gs; my %types_found = (); my @type_def = split(/\s*,\s*/, $str); foreach my $s (@type_def) { my $cur_type = ''; if ($s =~ /\s+OF\s+([^\s;]+)/) { $cur_type = $1; } elsif ($s =~ /\s+FROM\s+([^\s;]+)/) { $cur_type = uc($1); } elsif ($s =~ /^\s*([^\s]+)\s+([^\s]+)/) { $cur_type = $2; } push(@{$types_found{src_types}}, $cur_type); if (exists $all_types{$cur_type}) { push(@{$types_found{pg_types}}, $all_types{$cur_type}); } else { my $custom_type = $self->_get_types($cur_type); foreach my $tpe (sort {length($a->{name}) <=> length($b->{name}) } @{$custom_type}) { last if (uc($tpe->{name}) eq $cur_type); # prevent infinit loop $self->logit("\tLooking inside nested custom type $tpe->{name} to extract values...\n", 1); my %types_def = $self->_get_custom_types($tpe->{code}, $cur_type); if ($#{$types_def{pg_types}} >= 0) { $self->logit("\t\tfound subtype description: $tpe->{name}(" . join(',', @{$types_def{pg_types}}) . ")\n", 1); push(@{$types_found{pg_types}}, \@{$types_def{pg_types}}); push(@{$types_found{src_types}}, \@{$types_def{src_types}}); } } } } return %types_found; } sub format_data_row { my ($self, $row, $data_types, $action, $src_data_types, $custom_types, $table, $colcond, $sprep) = @_; @{ $self->{tables}{$table}{pk_where_clause} } = (); @{ $self->{tables}{$table}{lo_import_id} } = (); my $has_geom = 0; $has_geom = 1 if (grep(/^(SDO_GEOMETRY|ST_|STGEOM_)/, @$src_data_types)); for (my $idx = 0; $idx <= $#{$data_types}; $idx++) { my $data_type = $data_types->[$idx] || ''; if ($has_geom && $row->[$idx] && $src_data_types->[$idx] =~ /(SDO_GEOMETRY|ST_|STGEOM_)/i) { if ($self->{type} ne 'INSERT') { if (!$self->{is_mysql} && ($self->{geometry_extract_type} eq 'INTERNAL')) { use Ora2Pg::GEOM; my $geom_obj = new Ora2Pg::GEOM('srid' => $self->{spatial_srid}{$table}->[$idx]); $geom_obj->{geometry}{srid} = ''; $row->[$idx] = $geom_obj->parse_sdo_geometry($row->[$idx]) if ($row->[$idx] =~ /^ARRAY\(0x/); $row->[$idx] = 'SRID=' . $self->{spatial_srid}{$table}->[$idx] . ';' . $row->[$idx]; } elsif ($self->{geometry_extract_type} eq 'WKB') { if ($self->{is_mysql}) { $row->[$idx] =~ s/^SRID=(\d+);//; $self->{spatial_srid}{$table}->[$idx] = $1; } $row->[$idx] = unpack('H*', $row->[$idx]); $row->[$idx] = 'SRID=' . $self->{spatial_srid}{$table}->[$idx] . ';' . $row->[$idx]; } } elsif ($self->{geometry_extract_type} eq 'WKB') { if ($self->{is_mysql}) { $row->[$idx] =~ s/^SRID=(\d+);//; $self->{spatial_srid}{$table}->[$idx] = $1; } $row->[$idx] = unpack('H*', $row->[$idx]); $row->[$idx] = "'SRID=" . $self->{spatial_srid}{$table}->[$idx] . ';' . $row->[$idx] . "'"; } elsif (($self->{geometry_extract_type} eq 'INTERNAL') || ($self->{geometry_extract_type} eq 'WKT')) { if (!$self->{is_mysql}) { if ($src_data_types->[$idx] =~ /SDO_GEOMETRY/i) { use Ora2Pg::GEOM; my $geom_obj = new Ora2Pg::GEOM('srid' => $self->{spatial_srid}{$table}->[$idx]); $geom_obj->{geometry}{srid} = ''; $row->[$idx] = $geom_obj->parse_sdo_geometry($row->[$idx]) if ($row->[$idx] =~ /^ARRAY\(0x/); $row->[$idx] = "ST_GeomFromText('" . $row->[$idx] . "', " . $self->{spatial_srid}{$table}->[$idx] . ")"; } else { $row->[$idx] = "ST_Geometry('" . $row->[$idx] . "', $self->{spatial_srid}{$table}->[$idx])"; } } else { $row->[$idx] =~ s/^SRID=(\d+);//; $row->[$idx] = "ST_GeomFromText('" . $row->[$idx] . "', $1)"; } } # Stores the filter to use in the WHERE clause if (exists $self->{tables}{$table}{pk_columns}{$idx}) { push(@{ $self->{tables}{$table}{pk_where_clause} }, "$self->{tables}{$table}{pk_columns}{$idx} = $row->[$idx]"); } } elsif ($row->[$idx] =~ /^(?!(?!)\x{100})ARRAY\(0x/) { print STDERR "/!\\ WARNING /!\\: we should not be there !!!\n"; } else { $row->[$idx] = $self->format_data_type($row->[$idx], $data_type, $action, $table, $src_data_types->[$idx], $idx, $colcond->[$idx], $sprep); # Construct a WHERE clause based onb PK columns values if ($self->{lo_import} && $colcond->[$idx]->{isoid} && $colcond->[$idx]->{blob}) { # Store the uuid of the file containing the BLOB and set the oid to 0 if ($colcond->[$idx]->{isoid} && $colcond->[$idx]->{blob}) { push(@{ $self->{tables}{$table}{lo_import_id} }, $row->[$idx]); push(@{ $self->{tables}{$table}{lo_import_col} }, $self->{tables}{$table}{dest_column_name}[$idx]); $row->[$idx] = 0; } } # Stores the filter to use in the WHERE clause if (exists $self->{tables}{$table}{pk_columns}{$idx}) { push(@{ $self->{tables}{$table}{pk_where_clause} }, "$self->{tables}{$table}{pk_columns}{$idx} = $row->[$idx]"); } } } # Now add the script to import later the BLOB(s) into the table as a large object if ($self->{lo_import} && $#{ $self->{tables}{$table}{pk_where_clause} } >= 0) { # Rename table and double-quote it if required my $tmptb = $self->get_replaced_tbname($table); # Replace Tablename by temporary table for DATADIFF (data will be inserted in real table at the end) if ($self->{datadiff}) { $tmptb = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); } my $where_clause = join(' AND ', @{ $self->{tables}{$table}{pk_where_clause} }); # Generete the entry in the psql script foreach (my $i = 0; $i <= $#{ $self->{tables}{$table}{lo_import_id} }; $i++) { if ($self->{tables}{$table}{lo_import_id}[$i] ne '\N' && $self->{tables}{$table}{lo_import_id}[$i] ne 'NULL') { $self->{post_lo_script} .= qq{ ret=`psql -c "\\lo_import '$self->{tables}{$table}{lo_import_id}[$i]'" | awk '{print \$2}'` if [ "\${ret}a" != "a" ]; then psql -c "UPDATE $tmptb SET $self->{tables}{$table}{lo_import_col}[$i] = \${ret} WHERE $where_clause" fi }; } else { $self->{post_lo_script} .= qq{ psql -c "UPDATE $tmptb SET $self->{tables}{$table}{lo_import_col}[$i] = NULL WHERE $where_clause" }; } } } } sub set_custom_type_value { my ($self, $data_type, $user_type, $rows, $dest_type, $no_quote) = @_; my $has_array = 0; my @type_col = (); my $result = ''; my $col_ref = []; push(@$col_ref, @$rows); my $num_arr = -1; my $isnested = 0; for (my $i = 0; $i <= $#{$col_ref}; $i++) { if ($col_ref->[$i] !~ /^ARRAY\(0x/) { if ($self->{type} eq 'COPY') { # Want to export the user defined type as a single array, not composite type if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) { $has_array = 1; $col_ref->[$i] =~ s/"/\\\\"/gs; if ($col_ref->[$i] =~ /[,"]/) { $col_ref->[$i] = '"' . $col_ref->[$i] . '"'; }; # Data must be exported as an array of numeric types } elsif ($dest_type =~ /\[\d*\]$/) { $has_array = 1; } elsif ($dest_type =~ /(char|text)/) { $col_ref->[$i] =~ s/"/\\\\\\\\""/igs; if ($col_ref->[$i] =~ /[,"]/) { $col_ref->[$i] = '""' . $col_ref->[$i] . '""'; }; } else { $isnested = 1; } } else { # Want to export the user defined type as a single array, not composite type if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) { $has_array = 1; $col_ref->[$i] =~ s/"/\\"/gs; $col_ref->[$i] =~ s/'/''/gs; if ($col_ref->[$i] =~ /[,"]/) { $col_ref->[$i] = '"' . $col_ref->[$i] . '"'; }; # Data must be exported as a simple array of numeric types } elsif ($dest_type =~ /\[\d*\]$/i) { $has_array = 1; } elsif ($dest_type =~ /(char|text)/) { $col_ref->[$i] = "'" . $col_ref->[$i] . "'" if ($col_ref->[0][$i] ne ''); } else { $isnested = 1; } } push(@type_col, $col_ref->[$i]); } else { $num_arr++; my @arr_col = (); for (my $j = 0; $j <= $#{$col_ref->[$i]}; $j++) { # Look for data based on custom type to replace the reference by the value if ($col_ref->[$i][$j] =~ /^(?!(?!)\x{100})ARRAY\(0x/ && $user_type->{src_types}[$i][$j] !~ /SDO_GEOMETRY/i && $user_type->{src_types}[$i][$j] !~ /^(ST_|STGEOM_)/i #ArGis geometry types ) { my $dtype = uc($user_type->{src_types}[$i][$j]) || ''; $dtype =~ s/\(.*//; # remove any precision if (!exists $self->{data_type}{$dtype} && !exists $self->{user_type}{$dtype}) { %{ $self->{user_type}{$dtype} } = $self->custom_type_definition($dtype); } $col_ref->[$i][$j] = $self->set_custom_type_value($dtype, $self->{user_type}{$dtype}, $col_ref->[$i][$j], $user_type->{pg_types}[$i][$j], 1); if ($self->{type} ne 'COPY') { $col_ref->[$i][$j] =~ s/"/\\\\""/gs; } else { $col_ref->[$i][$j] =~ s/"/\\\\\\\\""/gs; } } if ($self->{type} eq 'COPY') { # Want to export the user defined type as charaters array if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) { $has_array = 1; $col_ref->[$i][$j] =~ s/"/\\\\"/gs; if ($col_ref->[$i][$j] =~ /[,"]/) { $col_ref->[$i][$j] = '"' . $col_ref->[$i][$j] . '"'; }; } # Data must be exported as an array of numeric types elsif ($dest_type =~ /\[\d*\]$/) { $has_array = 1; } } else { # Want to export the user defined type as array if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) { $has_array = 1; $col_ref->[$i][$j] =~ s/"/\\"/gs; $col_ref->[$i][$j] =~ s/'/''/gs; if ($col_ref->[$i][$j] =~ /[,"]/) { $col_ref->[$i][$j] = '"' . $col_ref->[$i][$j] . '"'; }; } # Data must be exported as an array of numeric types elsif ($dest_type =~ /\[\d*\]$/) { $has_array = 1; } } if ($col_ref->[$i][$j] =~ /[\(\)]/ && $col_ref->[$i][$j] !~ /^[\\]+""/) { if ($self->{type} ne 'COPY') { $col_ref->[$i][$j] = "\\\\\"\"" . $col_ref->[$i][$j] . "\\\\\"\""; } else { $col_ref->[$i][$j] = "\\\\\\\\\"\"" . $col_ref->[$i][$j] . "\\\\\\\\\"\""; } } push(@arr_col, $col_ref->[$i][$j]); } push(@type_col, '(' . join(',', @arr_col) . ')'); } } if ($has_array) { $result = '{' . join(',', @type_col) . '}'; } elsif ($isnested) { # ARRAY[ROW('B','C')] my $is_string = 0; foreach my $g (@{$self->{user_type}{$dest_type}->{pg_types}}) { $is_string = 1 if (grep(/(text|char|varying)/i, @$g)); } if ($is_string) { $result = '({"(' . join(',', @type_col) . ')"})'; } else { $result = '("{' . join(',', @type_col) . '}")'; } } else { # This is the root call of the function, no global quoting is required if (!$no_quote) { #map { s/^$/NULL/; } @type_col; #$result = 'ROW(ARRAY[ROW(' . join(',', @type_col) . ')])'; # With arrays of arrays the construction is different if ($num_arr > 1) { #### Expected # INSERT: '("{""(0,0,0,0,0,0,0,0,0,,,)"",""(0,0,0,0,0,0,0,0,0,,,)""}")' # COPY: ("{""(0,0,0,0,0,0,0,0,0,,,)"",""(0,0,0,0,0,0,0,0,0,,,)""}") #### $result = "(\"{\"\"" . join('"",""', @type_col) . "\"\"}\")"; } # When just one or none arrays are present else { #### Expected # INSERT: '("(1,1)",0,,)' # COPY: ("(1,1)",0,,) #### map { s/^\(([^\)]+)\)$/"($1)"/; } @type_col; $result = "(" . join(',', @type_col) . ")"; } # else we are in recusive call } else { $result = "\"(" . join(',', @type_col) . ")\""; } } if (!$no_quote && $self->{type} ne 'COPY') { $result = "'$result'"; } while ($result =~ s/,"""",/,NULL,/gs) {}; return $result; } sub format_data_type { my ($self, $col, $data_type, $action, $table, $src_type, $idx, $cond, $sprep, $isnested) = @_; my $q = "'"; $q = '"' if ($isnested); # Skip data type formatting when it has already been done in # set_custom_type_value(), aka when the data type is an array. next if ($data_type =~ /\[\d*\]/); # Internal timestamp retrieves from custom type is as follow: 01-JAN-77 12.00.00.000000 AM (internal_date_max) if (($data_type eq 'char') && $col =~ /^(\d{2})-([A-Z]{3})-(\d{2}) (\d{2})\.(\d{2})\.(\d{2}\.\d+) (AM|PM)$/ ) { my $d = $1; my $m = $ORACLE_MONTHS{$2}; my $y = $3; my $h = $4; my $min = $5; my $s = $6; my $typeh = $7; if ($typeh eq 'PM') { $h += 12; } if ($d <= $self->{internal_date_max}) { $d += 2000; } else { $d += 1900; } $col = "$y-$m-$d $h:$min:$s"; $data_type = 'timestamp'; $src_type = 'internal timestamp'; } # Workaround for a bug in DBD::Oracle with the ora_piece_lob option # (used when use_lob_locator is disabled) where null values fetch as # empty string for certain types. if (!$self->{use_lob_locator} and ($cond->{clob} or $cond->{blob} or $cond->{long})) { $col = undef if (!length($col)); } # Preparing data for output if ($action ne 'COPY') { if (!defined $col) { if (!$cond->{isnotnull} || ($self->{empty_lob_null} && ($cond->{clob} || $cond->{isbytea}))) { $col = 'NULL' if (!$sprep); } else { $col = "$q$q"; } } elsif ( ($src_type =~ /SDO_GEOMETRY/i) && ($self->{geometry_extract_type} eq 'WKB') ) { $col = "St_GeomFromWKB($q\\x" . unpack('H*', $col) . "$q, $self->{spatial_srid}{$table}->[$idx])"; } elsif ($cond->{isbytea} || ($self->{blob_to_lo} && $cond->{isoid} && $cond->{blob})) { $col = $self->_escape_lob($col, $cond->{raw} ? 'RAW' : 'BLOB', $cond, $isnested, $data_type); } elsif ($cond->{istext}) { if ($cond->{clob}) { $col = $self->_escape_lob($col, 'CLOB', $cond, $isnested, $data_type); } elsif (!$sprep) { $col = $self->escape_insert($col, $isnested); } } elsif ($cond->{isbit}) { $col = "B$q" . $col . "$q"; } elsif ($cond->{isdate}) { $q = '' if ( $col =~ /^['\`]/ ); if ($col =~ /^0000-/) { $col = $self->{replace_zero_date} ? "$q$self->{replace_zero_date}$q" : 'NULL'; } elsif ($col =~ /^(\d+-\d+-\d+ \d+:\d+:\d+)\.$/) { $col = "$q$1$q"; } else { $col = "$q$col$q"; } } elsif ($cond->{isboolean}) { if (exists $self->{ora_boolean_values}{lc($col)}) { $col = "$q" . $self->{ora_boolean_values}{lc($col)} . "$q"; } } elsif ($cond->{isefile}) { $col =~ s/\\/\\\\/g; $col =~ s/([\(\)])/\\$1/g; # escape comma except the first one $col =~ s/,/\,/g; $col =~ s/\,/,/; } elsif ($cond->{isinterval}) { if ($col =~ /^-/) { $col =~ s/ (\d+)/ -$1/; $col = "'$col'"; } } elsif ($cond->{isnum}) { if (!$self->{pg_dsn}) { $col =~ s/^([\-]*)(\~|Inf)$/'$1Infinity'/i; } else { $col =~ s/^([\-]*)(\~|Inf)$/$1Infinity/i; } } else { if (!$sprep) { $col = 'NULL' if ($col eq ''); } else { $col = undef if ($col eq ''); } } } else { if (!defined $col) { if (!$cond->{isnotnull} || ($self->{empty_lob_null} && ($cond->{clob} || $cond->{isbytea}))) { $col = '\N'; } else { $col = ''; } } elsif ( $cond->{geometry} && ($self->{geometry_extract_type} eq 'WKB') ) { $col = 'SRID=' . $self->{spatial_srid}{$table}->[$idx] . ';' . unpack('H*', $col); } elsif ($cond->{isboolean}) { if (exists $self->{ora_boolean_values}{lc($col)}) { $col = $self->{ora_boolean_values}{lc($col)}; } } elsif ($self->{lo_import} && $cond->{blob}) { # In copy mode we write the BLOB data to an external file # for later lo_import and we insert the corresponding unique # reference of this BLOB into the Oid column. The value of # the Oid will be fixed with the right Oid when importing # the large object. $col = $self->_save_blob_to_lo($col); } elsif ($cond->{isbytea}) { $col = $self->_escape_lob($col, $cond->{raw} ? 'RAW' : 'BLOB', $cond, $isnested, $data_type); } elsif ($cond->{isjson}) { # preserve json escaping $col =~ s/\\/\\\\/g; } elsif ($cond->{isinterval}) { if ($col =~ /^-/) { $col =~ s/ (\d+)/ -$1/; } } elsif ($cond->{isnum}) { $col =~ s/([\-]*)(\~|Inf)/$1Infinity/i; $col = '\N' if ($col eq ''); } elsif ($cond->{istext}) { $cond->{clob} ? $col = $self->_escape_lob($col, 'CLOB', $cond, $isnested) : $col = $self->escape_copy($col, $isnested, $data_type); } elsif ($cond->{isdate}) { if ($col =~ /^0000-/) { $col = $self->{replace_zero_date} || '\N'; } elsif ($col =~ /^(\d+-\d+-\d+ \d+:\d+:\d+)\.$/) { $col = $1; } } elsif ($cond->{isefile}) { $col =~ s/\\/\\\\/g; $col =~ s/([\(\)])/\\\\$1/g; # escape comma except the first one $col =~ s/,/\\,/g; $col =~ s/\\,/,/; } elsif ($cond->{isbit}) { $col = $col; } } return $col; } sub _save_blob_to_lo { my $self = shift; my $dirprefix = ''; $dirprefix = "$self->{output_dir}" if ($self->{output_dir}); my $filename = $dirprefix . &get_uuid() . '.lo'; $self->logit("DEBUG: write blob to $filename\n", 1); my $fh = new IO::File; $fh->open(">$filename") or $self->logit("FATAL: can not write $filename\n", 0, 1); $fh->binmode(':raw'); print $fh $_[0]; $fh->close(); return $filename; } sub get_uuid { my $uuid = ''; if (open(my $fh, "/proc/sys/kernel/random/uuid")) { $uuid = <$fh>; close($fh); } else { $uuid = `/usr/bin/uuidgen`; } chomp($uuid); return $uuid; } sub hs_cond { my ($self, $data_types, $src_data_types, $table) = @_; my $col_cond = []; for (my $idx = 0; $idx < scalar(@$data_types); $idx++) { my $hs={}; $hs->{geometry} = $src_data_types->[$idx] =~ /SDO_GEOMETRY/i ? 1 : 0; $hs->{isnum} = $data_types->[$idx] !~ /^(json|char|varchar|date|time|text|bytea|xml|uuid|citext|enum)/i ? 1 :0; $hs->{isdate} = $data_types->[$idx] =~ /^(date|time)/i ? 1 : 0; $hs->{raw} = $src_data_types->[$idx] =~ /RAW/i ? 1 : 0; $hs->{clob} = $src_data_types->[$idx] =~ /CLOB/i ? 1 : 0; $hs->{blob} = $src_data_types->[$idx] =~ /BLOB/i ? 1 : 0; $hs->{long} = $src_data_types->[$idx] =~ /LONG/i ? 1 : 0; $hs->{istext} = $data_types->[$idx] =~ /(char|text|xml|uuid|citext)/i ? 1 : 0; $hs->{isbytea} = $data_types->[$idx] =~ /bytea/i ? 1 : 0; $hs->{isoid} = $data_types->[$idx] =~ /oid/i ? 1 : 0; $hs->{isbit} = $data_types->[$idx] =~ /bit/i ? 1 : 0; $hs->{isboolean} = $data_types->[$idx] =~ /boolean/i ? 1 : 0; $hs->{isefile} = $data_types->[$idx] =~ /efile/i ? 1 : 0; $hs->{isinterval} = $data_types->[$idx] =~ /interval/i ? 1 : 0; $hs->{isnotnull} = 0; $hs->{isjson} = $data_types->[$idx] =~ /json/i ? 1 : 0; if ($self->{nullable}{$table}{$idx} =~ /^N/) { $hs->{isnotnull} = 1; } push @$col_cond, $hs; } return $col_cond; } sub format_data { my ($self, $rows, $data_types, $action, $src_data_types, $custom_types, $table) = @_; my $col_cond = $self->hs_cond($data_types,$src_data_types, $table); foreach my $row (@$rows) { $self->format_data_row($row,$data_types,$action,$src_data_types,$custom_types,$table,$col_cond); } if ($self->{post_lo_script}) { $self->append_lo_import_file($table); } } =head2 dump This function dump data to the right export output (gzip file, file or stdout). =cut sub dump { my ($self, $data, $fh) = @_; return if (!defined $data || $data eq ''); if (!$self->{compress}) { if (defined $fh) { $fh->print($data); } elsif (defined $self->{fhout}) { $self->{fhout}->print($data); } else { print $data; } } elsif ($self->{compress} eq 'Zlib') { if (not defined $fh) { $self->{fhout}->gzwrite($data) or $self->logit("FATAL: error dumping compressed data\n", 0, 1); } else { $fh->gzwrite($data) or $self->logit("FATAL: error dumping compressed data\n", 0, 1); } } elsif (defined $self->{fhout}) { $self->{fhout}->print($data); } else { $self->logit("FATAL: no filehandle to write output, this may not happen\n", 0, 1); } } =head2 data_dump This function dump data to the right output (gzip file, file or stdout) in multiprocess safety. File is open and locked before writing data, it is closed at end. =cut sub data_dump { my ($self, $data, $tname, $pname) = @_; return if ($self->{oracle_speed}); # get out of here if there is no data to dump return if (not defined $data or $data eq ''); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $filename = $self->{output}; my $rname = $tname; $rname .= '_' . $pname if (!$self->{rename_partition} && $pname); $rname = $pname if ($self->{rename_partition} && $pname); if ($self->{file_per_table}) { $filename = "${rname}_$self->{output}"; $filename = "tmp_$filename"; } # Set file temporary until the table export is done $self->logit("Dumping data from $rname to file: $filename\n", 1); if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { $self->close_export_file($self->{fhout}) if (defined $self->{fhout} && !$self->{file_per_table} && !$self->{pg_dsn}); my $fh = $self->append_export_file($filename); $self->set_binmode($fh) if (!$self->{compress}); flock($fh, 2) || die "FATAL: can't lock file $dirprefix$filename\n"; $fh->print($data); $self->close_export_file($fh); $self->logit("Written " . length($data) . " bytes to $dirprefix$filename\n", 1); # Reopen default output file $self->create_export_file() if (defined $self->{fhout} && !$self->{file_per_table} && !$self->{pg_dsn}); } elsif ($self->{file_per_table}) { if ($pname) { my $fh = $self->append_export_file($filename); $self->set_binmode($fh) if (!$self->{compress}); $fh->print($data); $self->close_export_file($fh); $self->logit("Written " . length($data) . " bytes to $dirprefix$filename\n", 1); } else { my $set_encoding = 0; if (!defined $self->{cfhout}) { $self->{cfhout} = $self->open_export_file($filename); $set_encoding = 1; } if ($self->{compress} eq 'Zlib') { $self->{cfhout}->gzwrite($data) or $self->logit("FATAL: error writing compressed data into $filename :: $self->{cfhout}\n", 0, 1); } else { $self->set_binmode($self->{cfhout}) if (!$self->{compress} && $set_encoding); $self->{cfhout}->print($data); } } } else { $self->dump($data); } } =head2 read_config This function read the specified configuration file. =cut sub read_config { my ($self, $file) = @_; my $fh = new IO::File; binmode($fh, ":utf8"); $fh->open($file) or $self->logit("FATAL: can't read configuration file $file, $!\n", 0, 1); while (my $l = <$fh>) { chomp($l); $l =~ s/\r//gs; $l =~ s/^\s*\#.*$//g; next if (!$l || ($l =~ /^\s+$/)); $l =~ s/^\s*//; $l =~ s/\s*$//; &parse_config($l); } $self->close_export_file($fh); } sub parse_config { my $l = shift; my ($var, $val) = split(/\s+/, $l, 2); $var = uc($var); if ($var eq 'IMPORT') { if ($val) { $self->logit("Importing $val...\n", 1); $self->read_config($val); $self->logit("Done importing $val.\n",1); } } elsif ($var =~ /^SKIP/) { if ($val) { $self->logit("No extraction of \L$val\E\n",1); my @skip = split(/[\s;,]+/, $val); foreach my $s (@skip) { $s = 'indexes' if ($s =~ /^indices$/i); $AConfig{"skip_\L$s\E"} = 1; } } } elsif ($var eq 'DEFINED_PK_' . $AConfig{SCHEMA}) # add schema specific definition of partitioning columns { my @defined_pk = split(/[\s;]+/, $val); foreach my $r (@defined_pk) { my ($table, $col) = split(/:/, $r); $AConfig{DEFINED_PK}{lc($table)} = $col; } } # Should be a else statement but keep the list up to date to memorize the directives full list elsif (!grep(/^$var$/, 'TABLES','ALLOW','MODIFY_STRUCT','REPLACE_TABLES','REPLACE_COLS', 'WHERE','EXCLUDE','VIEW_AS_TABLE','MVIEW_AS_TABLE','ORA_RESERVED_WORDS', 'SYSUSERS','REPLACE_AS_BOOLEAN','BOOLEAN_VALUES','MODIFY_TYPE','DEFINED_PK', 'ALLOW_PARTITION','REPLACE_QUERY','FKEY_ADD_UPDATE','DELETE', 'LOOK_FORWARD_FUNCTION','ORA_INITIAL_COMMAND','PG_INITIAL_COMMAND', 'TRANSFORM_VALUE','EXCLUDE_COLUMNS' )) { $AConfig{$var} = $val; if ($var eq 'NO_LOB_LOCATOR') { print STDERR "WARNING: NO_LOB_LOCATOR is deprecated, use USE_LOB_LOCATOR instead see documentation about the logic change.\n"; if ($val == 1) { $AConfig{USE_LOB_LOCATOR} = 0; } else { $AConfig{USE_LOB_LOCATOR} = 1; } } if ($var eq 'NO_BLOB_EXPORT') { print STDERR "WARNING: NO_BLOB_EXPORT is deprecated, use ENABLE_BLOB_EXPORT instead see documentation about the logic change.\n"; if ($val == 1) { $AConfig{ENABLE_BLOB_EXPORT} = 0; } else { $AConfig{ENABLE_BLOB_EXPORT} = 1; } } } elsif ($var =~ /VIEW_AS_TABLE/) { push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); } elsif ($var eq 'LOOK_FORWARD_FUNCTION') { push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); } elsif ( ($var eq 'TABLES') || ($var eq 'ALLOW') || ($var eq 'EXCLUDE') || ($var eq 'ALLOW_PARTITION') ) { $var = 'ALLOW' if ($var eq 'TABLES'); if ($var eq 'ALLOW_PARTITION') { $var = 'ALLOW'; push(@{$AConfig{$var}{PARTITION}}, split(/[,\s]+/, $val) ); } else { # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... # Global regex will be applied to the export type only my @vlist = split(/\s*;\s*/, $val); foreach my $a (@vlist) { if ($a =~ /^([^\[]+)\[(.*)\]$/) { push(@{$AConfig{$var}{"\U$1\E"}}, split(/[,\s]+/, $2) ); } else { push(@{$AConfig{$var}{ALL}}, split(/[,\s]+/, $a) ); } } } } elsif ( $var =~ /_INITIAL_COMMAND/ ) { push(@{$AConfig{$var}}, $val); } elsif ( $var eq 'SYSUSERS' ) { push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); } elsif ( $var eq 'ORA_RESERVED_WORDS' ) { push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); } elsif ( $var eq 'FKEY_ADD_UPDATE' ) { if (grep(/^$val$/i, @FKEY_OPTIONS)) { $AConfig{$var} = uc($val); } else { $self->logit("FATAL: invalid option, see FKEY_ADD_UPDATE in configuration file\n", 0, 1); } } elsif ($var eq 'MODIFY_STRUCT' or $var eq 'EXCLUDE_COLUMNS') { while ($val =~ s/([^\(\s]+)\s*\(([^\)]+)\)\s*//) { my $table = $1; my $fields = $2; $fields =~ s/^\s+//; $fields =~ s/\s+$//; push(@{$AConfig{$var}{$table}}, split(/[\s,]+/, $fields) ); } } elsif ($var eq 'MODIFY_TYPE') { $val =~ s/\\,/#NOSEP#/gs; my @modif_type = split(/[,;]+/, $val); foreach my $r (@modif_type) { $r =~ s/#NOSEP#/,/gs; my ($table, $col, $type) = split(/:/, lc($r)); $AConfig{$var}{$table}{$col} = $type; } } elsif ($var eq 'REPLACE_COLS') { while ($val =~ s/([^\(\s]+)\s*\(([^\)]+)\)[,;\s]*//) { my $table = $1; my $fields = $2; $fields =~ s/^\s+//; $fields =~ s/\s+$//; my @rel = split(/[,]+/, $fields); foreach my $r (@rel) { my ($old, $new) = split(/:/, $r); $AConfig{$var}{$table}{$old} = $new; } } } elsif ($var eq 'REPLACE_TABLES') { my @replace_tables = split(/[\s,;]+/, $val); foreach my $r (@replace_tables) { my ($old, $new) = split(/:/, $r); $AConfig{$var}{$old} = $new; } } elsif ($var eq 'REPLACE_AS_BOOLEAN') { my @replace_boolean = split(/[\s;]+/, $val); foreach my $r (@replace_boolean) { my ($table, $col) = split(/:/, $r); push(@{$AConfig{$var}{uc($table)}}, uc($col)); } } elsif ($var eq 'BOOLEAN_VALUES') { my @replace_boolean = split(/[\s,;]+/, $val); foreach my $r (@replace_boolean) { my ($yes, $no) = split(/:/, $r); $AConfig{$var}{lc($yes)} = 't'; $AConfig{$var}{lc($no)} = 'f'; } } elsif ($var eq 'DEFINED_PK') { my @defined_pk = split(/[\s;]+/, $val); foreach my $r (@defined_pk) { my ($table, $col) = split(/:/, $r); $AConfig{$var}{lc($table)} = $col; } } elsif ($var eq 'WHERE') { while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) { my $table = $1; my $where = $2; $where =~ s/^\s+//; $where =~ s/\s+$//; $AConfig{$var}{$table} = $where; } if ($val) { $AConfig{"GLOBAL_WHERE"} = $val; } } elsif ($var eq 'TRANSFORM_VALUE') { my @vals = split(/\s*;\s*/, $val); foreach my $v (@vals) { while ($v =~ s/([^\[\s]+)\s*\[\s*([^:,]+)\s*[,:]\s*([^\]]+)\s*\]\s*//) { my $table = $1; my $column = $2; my $clause = $3; $column =~ s/"//g; $AConfig{$var}{lc($table)}{lc($column)} = $clause; } } } elsif ($var eq 'DELETE') { while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) { my $table = $1; my $delete = $2; $delete =~ s/^\s+//; $delete =~ s/\s+$//; $AConfig{$var}{$table} = $delete; } if ($val) { $AConfig{"GLOBAL_DELETE"} = $val; } } elsif ($var eq 'REPLACE_QUERY') { while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) { my $table = lc($1); my $query = $2; $query =~ s/^\s+//; $query =~ s/\s+$//; $AConfig{$var}{$table} = $query; } } } sub _extract_functions { my ($self, $content) = @_; my @lines = split(/\n/s, $content); my @functions = (''); my $before = ''; my $fcname = ''; my $type = ''; for (my $i = 0; $i <= $#lines; $i++) { if ($lines[$i] =~ /^(?:CREATE|CREATE OR REPLACE)?\s*(?:NONEDITIONABLE|EDITIONABLE)?\s*(FUNCTION|PROCEDURE)\s+([a-z0-9_\$\-\."]+)(.*)/i) { $type = uc($1); $fcname = $2; my $after = $3; $fcname =~ s/^.*\.//; $fcname =~ s/"//g; $type = 'FUNCTION' if (!$self->{pg_supports_procedure}); if ($before) { push(@functions, "$before\n"); $functions[-1] .= "$type $fcname $after\n"; } else { push(@functions, "$type $fcname $after\n"); } $before = ''; } elsif ($fcname) { $functions[-1] .= "$lines[$i]\n"; } else { $before .= "$lines[$i]\n"; } $fcname = '' if ($lines[$i] =~ /^\s*END\s+["]*\Q$fcname\E["]*\b/i); } map { s/\bEND\s+(?!IF|LOOP|CASE|INTO|FROM|END|,)[a-z0-9_"\$]+\s*;/END;/igs; } @functions; return @functions; } =head2 _convert_package This function is used to rewrite Oracle PACKAGE code to PostgreSQL SCHEMA. Called only if PLSQL_PGSQL configuration directive is set to 1. =cut sub _convert_package { my ($self, $pkg) = @_; return if (!$pkg || !exists $self->{packages}{$pkg}); my $owner = $self->{packages}{$pkg}{owner} || ''; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $content = ''; if ($self->{package_as_schema}) { my $pname = $self->quote_object_name($pkg); $pname =~ s/^[^\.]+\.//; $content .= "\nDROP SCHEMA $self->{pg_supports_ifexists} $pname CASCADE;\n"; $content .= "CREATE SCHEMA IF NOT EXISTS $pname;\n"; if ($self->{force_owner}) { $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $content .= "ALTER SCHEMA \L$pname\E OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } } # Grab global declaration from the package header if ($self->{packages}{$pkg}{desc} =~ /CREATE OR REPLACE PACKAGE\s+([^\s]+)(?:\s*\%ORA2PG_COMMENT\d+\%)*\s*((?:AS|IS)(?:\s*\%ORA2PG_COMMENT\d+\%)*)\s*(.*)/is) { my $pname = $1; my $type = $2; my $glob_declare = $3; $pname =~ s/"//g; $pname =~ s/^.*\.//g; $self->logit("Looking global declaration in package $pname...\n", 1); # Process package spec to extract global variables $self->_remove_comments(\$glob_declare); # Remove multiline comment from declaration part while ($glob_declare =~ s/\%OPEN_COMMENT\%((?:.*)?\*\/)//s) {}; if ($glob_declare) { my @cursors = (); ($glob_declare, @cursors) = $self->clear_global_declaration($pname, $glob_declare, 0); # Then dump custom type foreach my $tpe (sort {$a->{pos} <=> $b->{pos}} @{$self->{types}}) { $self->logit("Dumping type $tpe->{name} from package description $pname...\n", 1); if ($self->{plsql_pgsql}) { $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}, %{$self->{pkg_type}{$pname}}); } else { if ($tpe->{code} !~ /^SUBTYPE\s+/i) { $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; } } $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; $content .= $tpe->{code} . "\n"; $i++; } $content .= join("\n", @cursors) . "\n"; $glob_declare = $self->register_global_variable($pname, $glob_declare); } @{$self->{types}} = (); } # Convert the package body part if ($self->{packages}{$pkg}{text} =~ /CREATE OR REPLACE PACKAGE\s+BODY\s*([^\s\%]+)(?:\s*\%ORA2PG_COMMENT\d+\%)*\s*(AS|IS)\s*(.*)/is) { my $pname = $1; my $type = $2; my $ctt = $3; my $glob_declare = $3; $pname =~ s/"//g; $pname =~ s/^.*\.//g; $self->logit("Dumping package $pname...\n", 1); # Process package spec to extract global variables $self->_remove_comments(\$glob_declare); if ($glob_declare && $glob_declare !~ /^(?:\s*\%ORA2PG_COMMENT\d+\%)*(FUNCTION|PROCEDURE)/is) { my @cursors = (); ($glob_declare, @cursors) = $self->clear_global_declaration($pname, $glob_declare, 1); # Then dump custom type foreach my $tpe (sort {$a->{pos} <=> $b->{pos}} @{$self->{types}}) { next if (!exists $self->{pkg_type}{$pname}{$tpe->{name}}); $self->logit("Dumping type $tpe->{name} from package body $pname...\n", 1); if ($self->{plsql_pgsql}) { $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}, %{$self->{pkg_type}{$pname}}); } else { if ($tpe->{code} !~ /^SUBTYPE\s+/i) { $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; } } $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; $content .= $tpe->{code} . "\n"; $i++; } $content .= join("\n", @cursors) . "\n"; $glob_declare = $self->register_global_variable($pname, $glob_declare); } if ($self->{file_per_function}) { my $dir = "$dirprefix".lc("$pname"); if (!-d "$dir") { if (not mkdir($dir)) { $self->logit("Fail creating directory package : $dir - $!\n", 1); next; } else { $self->logit("Creating directory package: $dir\n", 1); } } } $ctt =~ s/\bEND[^;]*;$//is; my @functions = $self->_extract_functions($ctt); # Try to detect local function for (my $i = 0; $i <= $#functions; $i++) { my %fct_detail = $self->_lookup_function($functions[$i], $pname); if (!exists $fct_detail{name}) { $functions[$i] = ''; next; } $fct_detail{name} =~ s/^.*\.//; $fct_detail{name} =~ s/"//g; next if (!$fct_detail{name}); $fct_detail{name} = lc($fct_detail{name}); if (!exists $self->{package_functions}{"\L$pname\E"}{$fct_detail{name}}) { my $res_name = $fct_detail{name}; $res_name =~ s/^[^\.]+\.//; $fct_detail{name} =~ s/^([^\.]+)\.//; if ($self->{package_as_schema}) { $res_name = $pname . '.' . $res_name; } else { $res_name = $pname . '_' . $res_name; } $res_name =~ s/"_"/_/g; $self->{package_functions}{"\L$pname\E"}{"\L$fct_detail{name}\E"}{name} = $self->quote_object_name($res_name); $self->{package_functions}{"\L$pname\E"}{"\L$fct_detail{name}\E"}{package} = $pname; } } $self->{pkgcost} = 0; foreach my $f (@functions) { next if (!$f); $content .= $self->_convert_function($owner, $f, $pkg || $pname); } if ($self->{estimate_cost}) { $self->{total_pkgcost} += $self->{pkgcost} || 0; } } @{$self->{types}} = (); return $content; } =head2 _restore_comments This function is used to restore comments into SQL code previously remove for easy parsing =cut sub _restore_comments { my ($self, $content) = @_; # Replace text values that was replaced in code $self->_restore_text_constant_part($content); # Restore comments while ($$content =~ /(\%ORA2PG_COMMENT\d+\%)[\n]*/is) { my $id = $1; my $sep = "\n"; # Do not append newline if this is a hint $sep = '' if ($self->{comment_values}{$id} =~ /^\/\*\+/); $$content =~ s/$id[\n]*/$self->{comment_values}{$id}$sep/is; delete $self->{comment_values}{$id}; }; # Restore start comment in a constant string $$content =~ s/\%OPEN_COMMENT\%/\/\*/gs; if ($self->{string_constant_regexp}) { # Replace potential text values that was replaced in comments $self->_restore_text_constant_part($content); } } =head2 _remove_comments This function is used to remove comments from SQL code to allow easy parsing =cut sub _remove_comments { my ($self, $content, $no_constant) = @_; # Fix comment in a string constant $$content = encode('UTF-8', $$content) if (!$self->{input_file} && $self->{force_plsql_encoding}); while ($$content =~ s/('[^';\n]*)\/\*([^';\n]*')/$1\%OPEN_COMMENT\%$2/s) {}; my %default_values = (); my $j = 0; while ($$content =~ s/(DEFAULT\s+)('[^']*')/$1\%DEFAULT$j\%/s) { $default_values{$j} = $2; $j++; }; # Fix unterminated comment at end of the code $$content =~ s/(\/\*(?:(?!\*\/).)*)$/$1 \*\//s; # multiline comment flags my $m_comment_flag = 'False'; # Replace some other cases that are breaking the parser (presence of -- in constant string, etc.) my @lines = split(/([\n\r]+)/, $$content); for (my $i = 0; $i <= $#lines; $i++) { next if ($lines[$i] !~ /\S/); # Fix mysql # comments if ($self->{is_mysql}) { $lines[$i] =~ s/^([\t ]*)#/$1--/; } if ($lines[$i] !~ /^[\t ]*\--.*\/\*.*\*\/.*$/ and $lines[$i] !~ /\/\*.*\*\/$/) { # Single line comment --...-- */ is replaced by */ only $lines[$i] =~ s/^([\t ]*)\-[\-]+\s*\*\//$1\*\//; # to check if we have starting multiline comment /* if (!($lines[$i] =~ /.*--.*\/\*/ and $lines[$i] !~ /.*\/\*.*--/)) { if ($lines[$i] =~ /\/\*.*$/ and $m_comment_flag eq 'False') { $m_comment_flag = 'True'; # setting flag to true } } # Check for -- and */ in the same line if ($lines[$i] =~ /(.*?--.*?)(\*\/.*)$/ and $m_comment_flag eq 'True') { $lines[$i] = $1; splice(@lines, $i + 1, 0, $2); $m_comment_flag = 'False'; } elsif ($lines[$i] =~ /(.*\*\/)/ and $m_comment_flag eq 'True') { $m_comment_flag = 'False'; } } # Single line comment -- if ($lines[$i] =~ s/^([\t ]*\-\-.*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; $self->{idxcomment}++; } # Single line comment /* ... */ if ($lines[$i] =~ s/^([\t ]*\/\*.*\*\/)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; $self->{idxcomment}++; } # ex: v := 'literal' -- commentaire avec un ' guillemet if ($lines[$i] =~ s/^([^']+'[^']*'\s*)(\-\-.*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; $self->{idxcomment}++; } # ex: ---/* REN 16.12.2010 ZKOUSKA TEST NA KOLURC if ($lines[$i] =~ s/^(\s*)(\-\-(?:(?!\*\/\s*$).)*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; $self->{idxcomment}++; } # ex: var1 := SUBSTR(var2,1,28) || ' -- ' || var3 || ' -- ' || SUBSTR(var4,1,26) ; while ($lines[$i] =~ s/('[^;']*\-\-[^']*')/\?TEXTVALUE$self->{text_values_pos}\?/) { $self->{text_values}{$self->{text_values_pos}} = $1; $self->{text_values_pos}++; } } $$content =join('', @lines); while ($$content =~ s/\%DEFAULT(\d+)\%/$default_values{$1}/s) {}; %default_values = (); # First remove hints they are not supported in PostgreSQL and it break the parser while ($$content =~ s/(\/\*\+(?:.*?)\*\/)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; $self->{idxcomment}++; } # Replace /* */ comments by a placeholder and save the comment while ($$content =~ s/(\/\*(.*?)\*\/)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; $self->{idxcomment}++; } while ($$content =~ s/(\'[^\'\n\r]+\b(PROCEDURE|FUNCTION)\s+[^\'\n\r]+\')/\%ORA2PG_COMMENT$self->{idxcomment}\%/is) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; $self->{idxcomment}++; } @lines = split(/\n/, $$content); for (my $j = 0; $j <= $#lines; $j++) { if (!$self->{is_mysql}) { # Extract multiline comments as a single placeholder my $old_j = $j; my $cmt = ''; while ($lines[$j] =~ /^(\s*\-\-.*)$/) { $cmt .= "$1\n"; $j++; } if ( $j > $old_j ) { chomp($cmt); $lines[$old_j] =~ s/^(\s*\-\-.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/; $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $cmt; $self->{idxcomment}++; $j--; while ($j > $old_j) { delete $lines[$j]; $j--; } } my $nocomment = ''; if ($lines[$j] =~ s/^([^']*)('[^\-\']*\-\-[^\-\']*')/$1\%NO_COMMENT\%/) { $nocomment = $2; } if ($lines[$j] =~ s/(\s*\-\-.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; chomp($self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"}); $self->{idxcomment}++; } $lines[$j] =~ s/\%NO_COMMENT\%/$nocomment/; } else { # Mysql supports differents kinds of comment's starter if ( ($lines[$j] =~ s/(\s*\-\- .*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) || (!grep(/^$self->{type}$/, 'FUNCTION', 'PROCEDURE') && $lines[$j] =~ s/(\s*COMMENT\s+'.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) || ($lines[$j] =~ s/(\s*\# .*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) ) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; chomp($self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"}); # Normalize start of comment $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} =~ s/^(\s*)COMMENT/$1\-\- /; $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} =~ s/^(\s*)\#/$1\-\- /; $self->{idxcomment}++; } } } $$content = join("\n", @lines); # Replace subsequent comment by a single one while ($$content =~ s/(\%ORA2PG_COMMENT\d+\%\s*\%ORA2PG_COMMENT\d+\%)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) { $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; $self->{idxcomment}++; } # Restore possible false positive constant replacement inside comment foreach my $k (keys %{ $self->{comment_values} } ) { $self->{comment_values}{$k} =~ s/\?TEXTVALUE(\d+)\?/$self->{text_values}{$1}/gs; } # Then replace text constant part to prevent a split on a ; or -- inside a text if (!$no_constant) { $self->_remove_text_constant_part($content); } } =head2 _convert_function This function is used to rewrite Oracle FUNCTION code to PostgreSQL. Called only if PLSQL_PGSQL configuration directive is set to 1. =cut sub _convert_function { my ($self, $owner, $plsql, $pname) = @_; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my %fct_detail = $self->_lookup_function($plsql, $pname); if ($self->{is_mysql}) { $pname = ''; } return if (!exists $fct_detail{name}); $fct_detail{name} =~ s/^.*\.// if ( (!$self->{is_mssql} || $self->{schema}) && (!$self->{input_file} && $self->{type} ne 'PACKAGE') ); $fct_detail{name} =~ s/"//gs; my $sep = '.'; $sep = '_' if (!$self->{package_as_schema}); my $fname = $self->quote_object_name($fct_detail{name}); $fname = $self->quote_object_name("$pname$sep$fct_detail{name}") if ($pname && !$self->{is_mysql} && !$self->{is_mssql}); $fname =~ s/"_"/_/gs; # rewrite argument syntax # Replace alternate syntax for default value $fct_detail{args} =~ s/:=/DEFAULT/igs; # NOCOPY not supported $fct_detail{args} =~ s/\s*NOCOPY//igs; # IN OUT should be INOUT $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; # Remove default IN keyword $fct_detail{args} =~ s/\s+IN\s+/ /igs; # Remove %ROWTYPE from arguments, we can use the table name as type $fct_detail{args} =~ s/\%ROWTYPE//igs; # Replace DEFAULT EMPTY_BLOB() from function/procedure arguments by DEFAULT NULL $fct_detail{args} =~ s/\s+DEFAULT\s+EMPTY_[CB]LOB\(\)/DEFAULT NULL/igs; # Input parameters after one with a default value must also have defaults # we add DEFAULT NULL to all remaining parameter without a default value. my @args_sorted = (); $fct_detail{args} =~ s/^\((.*)\)(\s*\%ORA2PG_COMMENT\d+\%)*\s*$/$1/gs; my $param_comments = $2 || ''; # Preserve parameters with precision and scale my $h = 0; my %param_param = (); while ($fct_detail{args} =~ s/\(([^\)]+)\)/%%tmp$h%%/s) { $param_param{$h} = $1; $h++; } if ($self->{use_default_null}) { my $has_default = 0; @args_sorted = split(',', $fct_detail{args}); for (my $i = 0; $i <= $#args_sorted; $i++) { $has_default = 1 if ($args_sorted[$i] =~ /\s+DEFAULT\s/i); if ($has_default && $args_sorted[$i] !~ /\s+DEFAULT\s/i) { # Add default null if this is not an OUT parameter if ( $args_sorted[$i] !~ /[,\(\s]OUT[\s,\)]/i && $args_sorted[$i] !~ /^OUT\s/i) { $args_sorted[$i] .= ' DEFAULT NULL'; } } } } else { # or we need to sort the arguments so the ones with default values will be on the bottom push(@args_sorted, grep {!/\sdefault\s/i} split ',', $fct_detail{args}); push(@args_sorted, grep {/\sdefault\s/i} split ',', $fct_detail{args}); my @orig_args = split(',', $fct_detail{args}); # Show a warning when there is parameters reordering my $fct_warning = ''; for (my $i = 0; $i <= $#args_sorted; $i++) { if ($args_sorted[$i] ne $orig_args[$i]) { my $str = $fct_detail{args}; $str =~ s/\%ORA2PG_COMMENT\d+\%//sg; $str =~ s/[\n\r]+//gs; $str =~ s/\s+/ /g; $self->_restore_text_constant_part(\$str); $fct_warning = "\n-- WARNING: parameters order has been changed by Ora2Pg to move parameters with default values at end\n"; $fct_warning .= "-- Original order was: $fname($str)\n"; $fct_warning .= "-- You will need to manually reorder parameters in the function calls\n"; print STDERR $fct_warning; last; } } } # Apply parameter list with translation for default values and reordering if needed for (my $i = 0; $i <= $#args_sorted; $i++) { if ($args_sorted[$i] =~ / DEFAULT ([^'].*)/i) { my $cod = Ora2Pg::PLSQL::convert_plsql_code($self, $1); $args_sorted[$i] =~ s/( DEFAULT )([^'].*)/$1$cod/i; } } $fct_detail{args} = '(' . join(',', @args_sorted) . ')'; $fct_detail{args} =~ s/\%\%tmp(\d+)\%\%/($param_param{$1})/gs; # Set the return part my $func_return = ''; $fct_detail{setof} = ' SETOF' if ($fct_detail{setof}); my $search_path = ''; if ($self->{export_schema} && !$self->{schema}) { $search_path = $self->set_search_path($owner); } # PostgreSQL procedure do not support OUT parameter, translate them into INOUT params if (!$fct_detail{hasreturn} && $self->{pg_supports_procedure} &&!$self->{pg_supports_outparam} && ($fct_detail{args} =~ /\bOUT\s+[^,\)]+/i)) { $fct_detail{args} =~ s/\bOUT(\s+[^,\)]+)/INOUT$1/igs; } my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; my $out_return = 0; if ($fct_detail{hasreturn}) { my $nbout = $#nout+1 + $#ninout+1; # When there is one or more out parameter, let PostgreSQL # choose the right type with not using a RETURNS clause. if ($nbout > 0) { $func_return = " AS \$body\$\n"; if ($fct_detail{type} ne 'PROCEDURE' || !$self->{PG_SUPPORTS_PROCEDURE}) { push(@nout, "extra_param $fct_detail{func_ret_type}"); $func_return = " RETURNS record AS \$body\$\n"; $out_return = 1; } $fct_detail{args} =~ s/\)$/, OUT extra_param $fct_detail{func_ret_type}\)/; } else { # Returns the right type $func_return = " RETURNS$fct_detail{setof} $fct_detail{func_ret_type} AS \$body\$\n"; } } elsif (!$self->{pg_supports_procedure}) { # Return void when there's no out parameters if (($#nout < 0) && ($#ninout < 0)) { $func_return = " RETURNS VOID AS \$body\$\n"; } else { # When there is one or more out parameter, let PostgreSQL # choose the right type with not using a RETURNS clause. $func_return = " AS \$body\$\n"; } } else { $func_return = " AS \$body\$\n"; } $func_return .= $param_comments; $func_return =~ s/\s+AS(\s+AS\s+)/$1/is; # extract custom type declared in a stored procedure my $create_type = ''; while ($fct_detail{declare} =~ s/\s+TYPE\s+([^\s]+)\s+IS\s+RECORD\s*\(([^;]+)\)\s*;//is) { my $tname = $1; my $tcode = $2; if ($pname && $self->{package_as_schema} && !$self->{is_mysql} && !$self->{is_mssql}) { $tname = $self->quote_object_name("$pname$sep$tname"); } else { $tname = $self->quote_object_name($tname); } $create_type .= "DROP TYPE $self->{pg_supports_ifexists} $tname;\n"; $create_type .= "CREATE TYPE $tname AS ($tcode);\n"; } while ($fct_detail{declare} =~ s/\s+TYPE\s+([^\s]+)\s+(AS|IS)\s*(VARRAY|VARYING ARRAY)\s*\((\d+)\)\s*OF\s*([^;]+);//is) { my $type_name = $1; my $size = $4; my $tbname = $5; $tbname =~ s/\s+NOT\s+NULL//g; chomp($tbname); $type_name =~ s/"//g; my $internal_name = $type_name; if ($pname && $self->{package_as_schema} && !$self->{is_mysql} && !$self->{is_mssql}) { $type_name = $self->quote_object_name("$pname$sep$type_name"); } elsif ($self->{export_schema} && !$self->{schema} && $owner) { $type_name = $self->quote_object_name("$owner.$type_name"); } $internal_name =~ s/^[^\.]+\.//; my $declar = $self->_replace_sql_type($tbname); $declar =~ s/[\n\r]+//s; $create_type .= "DROP TYPE $self->{pg_supports_ifexists} $type_name;\n"; $create_type .= "CREATE TYPE $type_name AS ($internal_name $declar\[$size\]);\n"; } my @at_ret_param = (); my @at_ret_type = (); my $at_suffix = ''; my $at_inout = 0; if ($fct_detail{declare} =~ s/\s*(PRAGMA\s+AUTONOMOUS_TRANSACTION[\s;]*)/-- $1/is && $self->{autonomous_transaction}) { $at_suffix = '_atx'; # COMMIT is not allowed in PLPGSQL function $fct_detail{code} =~ s/\bCOMMIT\s*;//; # Remove the pragma when a conversion is done $fct_detail{declare} =~ s/--\s+PRAGMA\s+AUTONOMOUS_TRANSACTION[\s;]*//is; my @tmp = split(',', $fct_detail{args}); $tmp[0] =~ s/^\(//; $tmp[-1] =~ s/\)$//; foreach my $p (@tmp) { if ($p =~ s/\bOUT\s+//) { $at_inout++; push(@at_ret_param, $p); push(@at_ret_type, $p); } elsif ($p =~ s/\bINOUT\s+//) { $at_inout++; push(@at_ret_param, $p); push(@at_ret_type, $p); } } map { s/^(.*?) //; } @at_ret_type; if ($fct_detail{hasreturn} && $#at_ret_param < 0) { push(@at_ret_param, 'ret ' . $fct_detail{func_ret_type}); push(@at_ret_type, $fct_detail{func_ret_type}); } map { s/^\s+//; } @at_ret_param; map { s/\s+$//; } @at_ret_param; map { s/^\s+//; } @at_ret_type; map { s/\s+$//; } @at_ret_type; } my $name = $fname; my $type = $fct_detail{type}; $type = 'FUNCTION' if (!$self->{pg_supports_procedure}); my $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type $fname$at_suffix $fct_detail{args}"; if (!$pname || !$self->{package_as_schema}) { if ($self->{export_schema} && !$self->{schema}) { if ($owner && !$self->{pg_schema}) { $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$owner.$fname$at_suffix") . " $fct_detail{args}"; $name = $self->quote_object_name("$owner.$fname"); $self->logit("Parsing function " . $self->quote_object_name("$owner.$fname") . "...\n", 1); } else { $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$fname$at_suffix") . " $fct_detail{args}"; $name = $self->quote_object_name("$fname"); $self->logit("Parsing function " . $self->quote_object_name("$fname") . "...\n", 1); } } elsif ($self->{export_schema} && $self->{schema}) { if (!$self->{pg_schema}) { $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$self->{schema}.$fname$at_suffix") . " $fct_detail{args}"; $name = $self->quote_object_name("$self->{schema}.$fname"); $self->logit("Parsing function " . $self->quote_object_name("$self->{schema}.$fname") . "...\n", 1); } else { $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$fname$at_suffix") . " $fct_detail{args}"; $name = $self->quote_object_name("$fname"); $self->logit("Parsing function " . $self->quote_object_name("$fname") . "...\n", 1); } } } else { $self->logit("Parsing function $fname...\n", 1); } # Create a wrapper for the function if we found an autonomous transaction my $at_wrapper = ''; if ($at_suffix && !$self->{pg_background}) { $at_wrapper = qq{ $search_path -- -- dblink wrapper to call function $name as an autonomous transaction -- CREATE EXTENSION IF NOT EXISTS dblink; }; $at_wrapper .= "CREATE$self->{create_or_replace} $type $name $fct_detail{args}$func_return"; my $params = ''; if ($#{$fct_detail{at_args}} >= 0) { map { s/(.+)/quote_nullable($1)/; } @{$fct_detail{at_args}}; $params = " ' || " . join(" || ',' || ", @{$fct_detail{at_args}}) . " || ' "; } my $dblink_conn = $self->{dblink_conn} || "'port=5432 dbname=testdb host=localhost user=pguser password=pgpass'"; $at_wrapper .= qq{DECLARE -- Change this to reflect the dblink connection string v_conn_str text := $dblink_conn; v_query text; }; my $call_str = 'SELECT * FROM'; $call_str = 'CALL' if (uc($type) eq 'PROCEDURE'); if ($#at_ret_param == 0) { my $varname = $at_ret_param[0]; $varname =~ s/\s+.*//; my $vartype = $at_ret_type[0]; $vartype =~ s/.*\s+//; if (!$fct_detail{hasreturn}) { $at_wrapper .= qq{ BEGIN v_query := 'CALL $fname$at_suffix ($params)'; SELECT v_ret INTO $varname FROM dblink(v_conn_str, v_query) AS p (v_ret $vartype); }; } else { $at_ret_type[0] = $fct_detail{func_ret_type}; $at_ret_param[0] = 'ret ' . $fct_detail{func_ret_type}; $at_wrapper .= qq{ v_ret $at_ret_type[0]; BEGIN v_query := 'SELECT * FROM $fname$at_suffix ($params)'; SELECT * INTO v_ret FROM dblink(v_conn_str, v_query) AS p ($at_ret_param[0]); RETURN v_ret; }; } } elsif ($#at_ret_param > 0) { my $varnames = ''; my $vartypes = ''; for (my $i = 0; $i <= $#at_ret_param; $i++) { my $v = $at_ret_param[$i]; $v =~ s/\s+.*//; $varnames .= "$v, "; $vartypes .= "v_ret$i "; my $t = $at_ret_type[$i]; $t =~ s/.*\s+//; $vartypes .= "$t, "; } $varnames =~ s/, $//; $vartypes =~ s/, $//; if (!$fct_detail{hasreturn}) { $at_wrapper .= qq{ BEGIN v_query := 'CALL $fname$at_suffix ($params)'; SELECT * FROM dblink(v_conn_str, v_query) AS p ($vartypes) INTO $varnames; }; } else { $at_ret_type[0] = $fct_detail{func_ret_type}; $at_ret_param[0] = 'ret ' . $fct_detail{func_ret_type}; $at_wrapper .= qq{ v_ret $at_ret_type[0]; BEGIN v_query := 'SELECT * FROM $fname$at_suffix ($params)'; SELECT * INTO v_ret FROM dblink(v_conn_str, v_query) AS p ($at_ret_param[0]); RETURN v_ret; }; } } elsif (!$fct_detail{hasreturn}) { $at_wrapper .= qq{ BEGIN v_query := 'CALL $fname$at_suffix ($params)'; PERFORM * FROM dblink(v_conn_str, v_query) AS p (ret boolean); }; } else { print STDERR "WARNING: we should not be there, please send the Oracle code of the $self->{type} to the author for debuging.\n"; } $at_wrapper .= qq{ END; \$body\$ LANGUAGE plpgsql SECURITY DEFINER; }; } elsif ($at_suffix && $self->{pg_background}) { $at_wrapper = qq{ $search_path -- -- pg_background wrapper to call function $name as an autonomous transaction -- CREATE EXTENSION IF NOT EXISTS pg_background; }; $at_wrapper .= "CREATE$self->{create_or_replace} $type $name $fct_detail{args}$func_return"; my $params = ''; if ($#{$fct_detail{at_args}} >= 0) { map { s/(.+)/quote_nullable($1)/; } @{$fct_detail{at_args}}; $params = " ' || " . join(" || ',' || ", @{$fct_detail{at_args}}) . " || ' "; } $at_wrapper .= qq{ DECLARE v_query text; }; if (!$fct_detail{hasreturn}) { $at_wrapper .= qq{ BEGIN v_query := 'SELECT true FROM $fname$at_suffix ($params)'; PERFORM * FROM pg_background_result(pg_background_launch(v_query)) AS p (ret boolean); }; } elsif ($#at_ret_param == 0) { my $prm = join(',', @at_ret_param); $at_wrapper .= qq{ v_ret $at_ret_type[0]; BEGIN v_query := 'SELECT * FROM $fname$at_suffix ($params)'; SELECT * INTO v_ret FROM pg_background_result(pg_background_launch(v_query)) AS p ($at_ret_param[0]); RETURN v_ret; }; } $at_wrapper .= qq{ END; \$body\$ LANGUAGE plpgsql SECURITY DEFINER; }; } # Add the return part of the function declaration $function .= $func_return; if ($fct_detail{immutable}) { $fct_detail{immutable} = ' IMMUTABLE'; } elsif ($plsql =~ /^FUNCTION/i) { # Oracle function can't modify data so always mark them as stable if ($self->{function_stable}) { $fct_detail{immutable} = ' STABLE'; } } if ($language && ($language !~ /SQL/i)) { $function .= "AS '$fct_detail{library}', '$fct_detail{library_fct}'\nLANGUAGE $language$fct_detail{immutable};\n"; $function =~ s/AS \$body\$//; } my $revoke = ''; if ($fct_detail{code}) { $fct_detail{declare} = '' if ($fct_detail{declare} !~ /[a-z]/is); $fct_detail{declare} =~ s/^\s*DECLARE//i; $fct_detail{declare} .= ';' if ($fct_detail{declare} && $fct_detail{declare} !~ /;\s*$/s && $fct_detail{declare} !~ /\%ORA2PG_COMMENT\d+\%\s*$/s); my $code_part = ''; $code_part .= "DECLARE\n$fct_detail{declare}\n" if ($fct_detail{declare}); $fct_detail{code} =~ s/^BEGIN\b//is; $code_part .= "BEGIN" . $fct_detail{code}; # Replace PL/SQL code into PL/PGSQL similar code $function .= Ora2Pg::PLSQL::convert_plsql_code($self, $code_part); $function .= ';' if ($function !~ /END\s*;\s*$/is && $fct_detail{code} !~ /\%ORA2PG_COMMENT\d+\%\s*$/); $function .= "\n\$body\$\nLANGUAGE PLPGSQL\n"; # Fix RETURN call when the function has OUT parameters if ($out_return) { $self->_remove_text_constant_part(\$function); $function =~ s/(\s+)RETURN\s*(\([^;]+\))\s*;/$1extra_param := $2;$1RETURN;/igs; $function =~ s/(\s+)RETURN\s+([^;]+);/$1extra_param := $2;$1RETURN;/igs; $self->_restore_text_constant_part(\$function); } $revoke = "-- REVOKE ALL ON $type $name $fct_detail{args} FROM PUBLIC;"; if ($at_suffix) { $revoke .= "\n-- REVOKE ALL ON $type $name$at_suffix $fct_detail{args} FROM PUBLIC;"; } $revoke =~ s/[\n\r]+\s*/ /gs; $revoke .= "\n"; if ($self->{force_security_invoker}) { $function .= "SECURITY INVOKER\n"; } else { if ($self->{type} ne 'PACKAGE') { if (!$self->{is_mysql}) { # A SECURITY DEFINER procedure cannot execute transaction control statements $function .= "SECURITY DEFINER\n" if ($self->{security}{"\U$fct_detail{name}\E"}{security} eq 'DEFINER' && $fct_detail{code} !~ /\b(COMMIT|ROLLBACK)\s*;/i); } else { $function .= "SECURITY DEFINER\n" if ($fct_detail{security} eq 'DEFINER'); } } else { # A SECURITY DEFINER procedure cannot execute transaction control statements $function .= "SECURITY DEFINER\n" if ($self->{security}{"\U$pname\E"}{security} eq 'DEFINER' && $fct_detail{code} !~ /\b(COMMIT|ROLLBACK)\s*;/i); } } $fct_detail{immutable} = '' if ($fct_detail{code} =~ /\b(UPDATE|INSERT|DELETE|CALL)\b/is); $function .= "$fct_detail{immutable};\n"; $function = "\n$fct_detail{before}$function"; } if ($self->{force_owner}) { $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $function .= "ALTER $type $name $fct_detail{args} OWNER TO"; $function .= " " . $self->quote_object_name($owner) . ";\n"; if ($at_suffix) { $function .= "ALTER $type $name$at_suffix $fct_detail{args} OWNER TO"; $function .= " " . $self->quote_object_name($owner) . ";\n"; } } } my $act_type = $type; $act_type = 'FUNCTION' if (!$self->{pg_supports_procedure}); $function .= "\nCOMMENT ON $act_type $name$at_suffix $fct_detail{args} IS $fct_detail{comment};\n" if ($fct_detail{comment}); $function .= $revoke; $function = $at_wrapper . $function; $fname =~ s/"//g; # Remove case sensitivity quoting $fname =~ s/^$pname\.//i; # remove package name if ($pname && $self->{file_per_function}) { $self->logit("\tDumping to one file per function: $dirprefix\L$pname/$fname\E_$self->{output}\n", 1); my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; $sql_header .= "-- Copyright 2000-2025 Gilles DAROLD. All rights reserved.\n"; $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; if ($self->{client_encoding}) { $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n"; } $sql_header .= $self->set_search_path(undef, $pname); $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); $sql_header = '' if ($self->{no_header}); if ($self->{print_dependencies} && $self->{plsql_pgsql} && !$self->{no_function_metadata}) { # look for other routines call in the stored procedure foreach my $sch (sort keys %{ $self->{function_metadata} }) { foreach my $pkg_name (sort keys %{ $self->{function_metadata}{$sch} }) { foreach my $fct_name (sort keys %{ $self->{function_metadata}{$sch}{$pkg_name} }) { next if ($fct_name =~ /^$fname$/i || $fct_name =~ /^.*\.$fname$/i); if ($fct_detail{code} =~ /\b$fct_name\b/is) { push(@{ $self->{object_dependencies}{uc("$pname.$fname")}{routines} }, uc("$sch.$fct_name")); } } } } # Look for merge/insert/update/delete @{ $self->{object_dependencies}{uc("$pname.$fname")}{merge} } = $function =~ /\bMERGE\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$pname.$fname")}{insert} } = $function =~ /\bINSERT\s+INTO\s+([^\(\s;,]+)/igs; @{ $self->{object_dependencies}{uc("$pname.$fname")}{update} } = $function =~ /(?:(?!FOR).)*?\s*\bUPDATE\s+([^\s;,]+)\s+/igs; @{ $self->{object_dependencies}{uc("$pname.$fname")}{delete} } = $function =~ /\b(?:DELETE\s+FROM|TRUNCATE\s+TABLE)\s+([^\s;,]+)\s+/igs; } my $fhdl = $self->open_export_file("$dirprefix\L$pname/$fname\E_$self->{output}", 1); $self->set_binmode($fhdl) if (!$self->{compress}); $self->_restore_comments(\$function); $self->normalize_function_call(\$function); $function =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; $self->dump($sql_header . $function, $fhdl); $self->close_export_file($fhdl); my $f = "$dirprefix\L$pname/$fname\E_$self->{output}"; $f = "\L$pname/$fname\E_$self->{output}" if ($self->{psql_relative_path}); $f =~ s/\.(?:gz|bz2)$//i; $function = "\\i$self->{psql_relative_path} $f\n"; $self->save_filetoupdate_list(lc($pname), lc($fname), "$dirprefix\L$pname/$fname\E_$self->{output}"); return $function; } elsif ($pname) { $self->save_filetoupdate_list(lc($pname), lc($fname), "$dirprefix$self->{output}"); } $function =~ s/\r//gs; $function =~ s/\bEND[\s;]+;/END;/is; my @lines = split(/\n/, $function); map { s/^\/$//; } @lines; return join("\n", @lines); } =head2 _convert_declare This function is used to rewrite Oracle FUNCTION declaration code to PostgreSQL. Called only if PLSQL_PGSQL configuration directive is set to 1. =cut sub _convert_declare { my ($self, $declare) = @_; $declare =~ s/\s+$//s; return if (!$declare); my @allwithcomments = split(/(\%ORA2PG_COMMENT\d+\%\n*)/s, $declare); for (my $i = 0; $i <= $#allwithcomments; $i++) { next if ($allwithcomments[$i] =~ /ORA2PG_COMMENT/); my @pg_declare = (); foreach my $tmp_var (split(/;/,$allwithcomments[$i])) { # Not cursor declaration if ($tmp_var !~ /\bcursor\b/is) { # Extract default assignment my $tmp_assign = ''; if ($tmp_var =~ s/\s*(:=|DEFAULT)(.*)$//is) { $tmp_assign = " $1$2"; } # Extract variable name and type my $tmp_pref = ''; my $tmp_name = ''; my $tmp_type = ''; if ($tmp_var =~ /(\s*)([^\s]+)\s+(.*?)$/s) { $tmp_pref = $1; $tmp_name = $2; $tmp_type = $3; $tmp_type =~ s/\s+//gs; if ($tmp_type =~ /([^\(]+)\(([^\)]+)\)/) { my $type_name = $1; my ($prec, $scale) = split(/,/, $2); $scale ||= 0; my $len = $prec; $prec = 0 if (!$scale); $len =~ s/\D//g; $tmp_type = $self->_sql_type($type_name,$len,$prec,$scale,$tmp_assign); } else { $tmp_type = $self->_sql_type($tmp_type); } push(@pg_declare, "$tmp_pref$tmp_name $tmp_type$tmp_assign;"); } } else { push(@pg_declare, "$tmp_var;"); } } $allwithcomments[$i] = join("", @pg_declare); } return join("", @allwithcomments); } =head2 _format_view This function is used to rewrite Oracle VIEW declaration code to PostgreSQL. =cut sub _format_view { my ($self, $view, $sqlstr) = @_; $self->_remove_comments(\$sqlstr); # Retrieve the column part of the view to remove double quotes if (!$self->{preserve_case} && $sqlstr =~ s/^(.*?)\bFROM\b/FROM/is) { my $tmp = $1; $tmp =~ s/"//gs; $sqlstr = $tmp . $sqlstr; } my @tbs = (); # Retrieve all tbs names used in view if possible if ($sqlstr =~ /\bFROM\b(.*)/is) { my $tmp = $1; $tmp =~ s/\%ORA2PG_COMMENT\d+\%//gs; $tmp =~ s/\s+/ /gs; $tmp =~ s/\bWHERE.*//is; # Remove all SQL reserved words of FROM STATEMENT $tmp =~ s/(LEFT|RIGHT|INNER|OUTER|NATURAL|CROSS|JOIN|\(|\))//igs; # Remove all ON join, if any $tmp =~ s/\bON\b[A-Z_\.\s]*=[A-Z_\.\s]*//igs; # Sub , with whitespace $tmp =~ s/,/ /g; my @tmp_tbs = split(/\s+/, $tmp); foreach my $p (@tmp_tbs) { push(@tbs, $p) if ($p =~ /^[A-Z_0-9\$]+$/i); } } foreach my $tb (@tbs) { next if (!$tb); my $regextb = $tb; $regextb =~ s/\$/\\\$/g; if (!$self->{preserve_case}) { # Escape column name $sqlstr =~ s/["']*\b$regextb\b["']*\.["']*([A-Z_0-9\$]+)["']*(,?)/$tb.$1$2/igs; # Escape table name $sqlstr =~ s/(^=\s?)["']*\b$regextb\b["']*/$tb/igs; } else { # Escape column name $sqlstr =~ s/["']*\b${regextb}["']*\.["']*([A-Z_0-9\$]+)["']*(,?)/"$tb"."$1"$2/igs; # Escape table name $sqlstr =~ s/(^=\s?)["']*\b$regextb\b["']*/"$tb"/igs; if ($tb =~ /(.*)\.(.*)/) { my $prefx = $1; my $sufx = $2; $sqlstr =~ s/"$regextb"/"$prefx"\."$sufx/g; } } } # replace column name in view query definition if needed foreach my $c (sort { $b cmp $a } keys %{ $self->{replaced_cols}{"\L$view\E"} }) { my $nm = $self->{replaced_cols}{"\L$view\E"}{$c}; $sqlstr =~ s/([\(,\s\."])$c([,\s\.:"\)])/$1$nm$2/ig; } if ($self->{plsql_pgsql}) { $sqlstr = Ora2Pg::PLSQL::convert_plsql_code($self, $sqlstr); } $self->_restore_comments(\$sqlstr); return $sqlstr; } =head2 randpattern This function is used to replace the use of perl module String::Random and is simply a cut & paste from this module. =cut sub randpattern { my $patt = shift; my $string = ''; my @upper=("A".."Z"); my @lower=("a".."z"); my @digit=("0".."9"); my %patterns = ( 'C' => [ @upper ], 'c' => [ @lower ], 'n' => [ @digit ], ); for my $ch (split(//, $patt)) { if (exists $patterns{$ch}) { $string .= $patterns{$ch}->[int(rand(scalar(@{$patterns{$ch}})))]; } else { $string .= $ch; } } return $string; } =head2 logit This function log information to STDOUT or to a logfile following a debug level. If critical is set, it dies after writing to log. =cut sub logit { my ($self, $message, $level, $critical) = @_; # Assessment report are dumped to stdin so avoid printing debug info return if (!$critical && $self->{type} eq 'SHOW_REPORT'); $level ||= 0; $message = '[' . strftime("%Y-%m-%d %H:%M:%S", localtime(time)) . '] ' . $message if ($self->{debug}); if ($self->{debug} >= $level) { if (defined $self->{fhlog}) { $self->{fhlog}->print($message); } else { print $message; } } if ($critical) { if ($self->{debug} < $level) { if (defined $self->{fhlog}) { $self->{fhlog}->print($message); } else { print "$message\n"; } } $self->{fhlog}->close() if (defined $self->{fhlog}); $self->{dbh}->disconnect() if ($self->{dbh}); $self->{dbhdest}->disconnect() if ($self->{dbhdest}); die "Aborting export...\n"; } } =head2 logrep This function log report's information to STDOUT or to a logfile. =cut sub logrep { my ($self, $message) = @_; if (defined $self->{fhlog}) { $self->{fhlog}->print($message); } else { print $message; } } =head2 _convert_type This function is used to rewrite Oracle TYPE DDL =cut sub _convert_type { my ($self, $plsql, $owner, %pkg_type) = @_; my ($package, $filename, $line) = caller; my $unsupported = "-- Unsupported, please edit to match PostgreSQL syntax\n"; my $content = ''; my $type_name = ''; $plsql =~ s/AUTHID DEFINER//is; # Replace SUBTYPE declaration into DOMAIN declaration if ($plsql =~ s/SUBTYPE\s+/CREATE DOMAIN /i) { $plsql =~ s/\s+IS\s+/ AS /; $plsql =~ s/^CREATE TYPE/TYPE/i; $plsql = $self->_replace_sql_type($plsql); return $plsql; } $plsql =~ s/\s*INDEX\s+BY\s+([^\s;]+)//is; $plsql =~ s/TYPE BODY \w+ AS.+?\nEND;//is; # Remove BODY $plsql =~ s/ +, CONSTRUCTOR FUNCTION [^\n]+ RETURN self AS RESULT\n//i; # Remove Constructor if ($plsql =~ /TYPE\s+([^\s]+)\s+(IS|AS)\s+TABLE\s+OF\s+(.*)/is) { $type_name = $1; my $type_of = $3; $type_name =~ s/"//g; if ($self->{export_schema} && !$self->{schema} && $owner && $type_name !~ /\./) { $type_name = "$owner.$type_name"; } my $internal_name = $type_name; $internal_name =~ s/^[^\.]+\.//; $type_of =~ s/\s*\(\s*/\(/; $type_of =~ s/\s*\)\s*/\)/; $type_of =~ s/\s*NOT[\t\s]+NULL//is; $type_of =~ s/\s*;\s*$//s; $type_of =~ s/^\s+//s; if ($type_of !~ /\s/s || $type_of =~ /VARCHAR2\(\d+ (CHAR|BYTE)\)/ # workaround for VARCHAR2 with type ) { $type_of = $self->_replace_sql_type($type_of); $self->{type_of_type}{'Nested Tables'}++; $content .= "DROP TYPE $self->{pg_supports_ifexists} " . $self->quote_object_name($type_name) . ";\n" if ($self->{drop_if_exists}); $content .= "CREATE TYPE " . $self->quote_object_name($type_name) . " AS (" . $self->quote_object_name($internal_name) . " $type_of\[\]);\n"; } else { $self->{type_of_type}{'Associative Arrays'}++; $self->logit("WARNING: this kind of Nested Tables are not supported, skipping type $1\n", 1); return "${unsupported}CREATE$self->{create_or_replace} $plsql"; } } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*REF\s+CURSOR/is) { $self->logit("WARNING: TYPE REF CURSOR are not supported, skipping type $1\n", 1); $plsql =~ s/\bREF\s+CURSOR/REFCURSOR/is; $self->{type_of_type}{'Type Ref Cursor'}++; return "${unsupported}CREATE$self->{create_or_replace} $plsql"; } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*OBJECT\s*\((.*?)(TYPE BODY.*)/is) { $self->{type_of_type}{'Type Boby'}++; $self->logit("WARNING: TYPE BODY are not supported, skipping type $1\n", 1); return "${unsupported}CREATE$self->{create_or_replace} $plsql"; } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*(?:OBJECT|RECORD)\s*\((.*)\)([^\)]*)/is) { $type_name = $1; my $description = $3; my $notfinal = $4; $notfinal =~ s/\s+/ /gs; if ($description =~ /\s*(MAP MEMBER|MEMBER|CONSTRUCTOR)\s+(FUNCTION|PROCEDURE).*/is) { $self->{type_of_type}{'Type with member method'}++; $self->logit("WARNING: TYPE with CONSTRUCTOR and MEMBER FUNCTION are not supported, skipping type $type_name\n", 1); return "${unsupported}CREATE$self->{create_or_replace} $plsql"; } $description =~ s/^\s+//s; my $declar = $self->_replace_sql_type($description); $type_name =~ s/"//g; if ($self->{export_schema} && !$self->{schema} && $owner && $type_name !~ /\./) { $type_name = "$owner.$type_name"; } if ($notfinal =~ /FINAL/is) { $content = "-- Inherited types are not supported in PostgreSQL, replacing with inherited table\n"; $content .= "CREATE TABLE " . $self->quote_object_name($type_name) . qq{ ( $declar ); }; $self->{type_of_type}{'Type inherited'}++; } else { $content = "DROP TYPE $self->{pg_supports_ifexists} " . $self->quote_object_name($type_name) . ";\n" if ($self->{drop_if_exists}); # add optional DROP $content .= "CREATE TYPE " . $self->quote_object_name($type_name) . " AS ("; $content .= qq{ $declar ); }; $self->{type_of_type}{'Object type'}++; } } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+UNDER\s*([^\s]+)\s+\((.*)\)([^\)]*)/is) { $type_name = $1; my $type_inherit = $2; my $description = $3; if ($description =~ /\s*(MAP MEMBER|MEMBER|CONSTRUCTOR)\s+(FUNCTION|PROCEDURE).*/is) { $self->logit("WARNING: TYPE with CONSTRUCTOR and MEMBER FUNCTION are not supported, skipping type $type_name\n", 1); $self->{type_of_type}{'Type with member method'}++; return "${unsupported}CREATE$self->{create_or_replace} $plsql"; } $description =~ s/^\s+//s; my $declar = $self->_replace_sql_type($description); $type_name =~ s/"//g; if ($self->{export_schema} && !$self->{schema} && $owner && $type_name !~ /\./) { $type_name = "$owner.$type_name"; } $content = "CREATE TABLE " . $self->quote_object_name($type_name) . " ("; $content .= qq{ $declar ) INHERITS (\L$type_inherit\E); }; $self->{type_of_type}{'Subtype'}++; } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*(VARRAY|VARYING ARRAY)\s*\((\d+)\)\s*OF\s*(.*)/is) { $type_name = $1; my $size = $4; my $tbname = $5; $type_name =~ s/"//g; if ($self->{export_schema} && !$self->{schema} && $owner && $type_name !~ /\./) { $type_name = "$owner.$type_name"; } $tbname =~ s/;//g; $tbname =~ s/\s+NOT\s+NULL//g; my $internal_name = $type_name; chomp($tbname); $internal_name =~ s/^[^\.]+\.//; my $declar = $self->_replace_sql_type($tbname); $declar =~ s/[\n\r]+//s; $content = "CREATE TYPE " . $self->quote_object_name($type_name) . " AS (" . $self->quote_object_name($internal_name) . " $declar\[$size\]);\n"; $self->{type_of_type}{Varrays}++; } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+FROM\s+(.*)( NOT NULL)?;/is) { my $typname = $1; my $notnull = $3; my $dtype = $self->_replace_sql_type($2); $content .= "CREATE DOMAIN $typname AS $dtype$notnull;\n"; } else { $self->{type_of_type}{Unknown}++; $plsql =~ s/;$//s; $content = "${unsupported}CREATE$self->{create_or_replace} $plsql;" } if ($self->{force_owner}) { $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); if ($owner) { $content .= "ALTER TYPE " . $self->quote_object_name($type_name) . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; } } # Prefix type with their own package name foreach my $t (keys %pkg_type) { $content =~ s/(\s+)($t)\b/$1$pkg_type{$2}/igs; } return $content; } sub ask_for_data { my ($self, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $is_subpart) = @_; # Build SQL query to retrieve data from this table if (!$part_name) { $self->logit("Looking how to retrieve data from $table...\n", 1); } elsif ($is_subpart) { $self->logit("Looking how to retrieve data from $table subpartition $part_name...\n", 1); } else { $self->logit("Looking how to retrieve data from $table partition $part_name...\n", 1); } my $query = $self->_howto_get_data($table, $nn, $tt, $stt, $part_name, $is_subpart); # Query with no column if (!$query) { $self->logit("WARNING: can not extract data from $table, no column found...\n", 0); return 0; } # Check for boolean rewritting for (my $i = 0; $i <= $#{$nn}; $i++) { my $colname = $nn->[$i]->[0]; $colname =~ s/["`]//g; my $typlen = $nn->[$i]->[5]; $typlen ||= $nn->[$i]->[2]; # Check if this column should be replaced by a boolean following table/column name if (grep(/^$colname$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { $tt->[$i] = 'boolean'; } # Check if this column should be replaced by a boolean following type/precision elsif (exists $self->{'replace_as_boolean'}{uc($nn->[$i]->[1])} && ($self->{'replace_as_boolean'}{uc($nn->[$i]->[1])}[0] == $typlen)) { $tt->[$i] = 'boolean'; } } # check if destination column type must be changed for (my $i = 0; $i <= $#{$nn}; $i++) { my $colname = $nn->[$i]->[0]; $colname =~ s/["`]//g; $tt->[$i] = $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"} if (exists $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"}); } # Look for user defined type if (!$self->{is_mysql}) { for (my $idx = 0; $idx < scalar(@$stt); $idx++) { my $data_type = uc($stt->[$idx]) || ''; $data_type =~ s/\(.*//; # remove any precision # in case of user defined type try to gather the underlying base types if (!exists $self->{data_type}{$data_type} && !exists $self->{user_type}{$data_type} && $data_type !~ /SDO_GEOMETRY/i && $data_type !~ /^(ST_|STGEOM_)/i #ArGis geometry types ) { %{ $self->{user_type}{$data_type} } = $self->custom_type_definition($data_type); } } } if ( ($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"} ) { $self->{ora_conn_count} = 0; while ($self->{ora_conn_count} < $self->{oracle_copies}) { spawn sub { $self->logit("Creating new connection to database to extract data...\n", 1); $self->_extract_data($query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $self->{ora_conn_count}); }; $self->{ora_conn_count}++; } # Wait for oracle connection terminaison while ($self->{ora_conn_count} > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{ora_conn_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } if (defined $pipe) { my $t_name = $part_name || $table; my $t_time = time(); $pipe->print("TABLE EXPORT ENDED: $t_name, end: $t_time, report all parts\n"); } } else { my $total_record = $self->_extract_data($query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name); # Only useful for single process return $total_record; } return; } sub custom_type_definition { my ($self, $custom_type, $parent, $is_nested) = @_; my %user_type = (); my $orig = $custom_type; my $data_type = uc($custom_type) || ''; $data_type =~ s/\(.*//; # remove any precision if (!exists $self->{data_type}{$data_type}) { if (!$is_nested) { $self->logit("Data type $custom_type is not native, searching on custom types.\n", 1); } else { $self->logit("\tData type $custom_type nested from type $parent is not native, searching on custom types.\n", 1); } $custom_type = $self->_get_types($custom_type); foreach my $tpe (sort {length($a->{name}) <=> length($b->{name}) } @{$custom_type}) { $self->logit("\tLooking inside custom type $tpe->{name} to extract values...\n", 1); my %types_def = $self->_get_custom_types($tpe->{code}); if ($#{$types_def{pg_types}} >= 0) { $self->logit("\tfound type description: $tpe->{name}(" . join(',', @{$types_def{pg_types}}) . ")\n", 1); push(@{$user_type{pg_types}} , \@{$types_def{pg_types}}); push(@{$user_type{src_types}}, \@{$types_def{src_types}}); } else { if ($tpe->{code} =~ /AS\s+VARRAY\s*(.*?)\s+OF\s+([^\s;]+);/is) { return $self->custom_type_definition(uc($2), $orig, 1); } elsif ($tpe->{code} =~ /(.*FROM\s+[^;\s\(]+)/is) { %types_def = $self->_get_custom_types($1); push(@{$user_type{pg_types}} , \@{$types_def{pg_types}}); push(@{$user_type{src_types}}, \@{$types_def{src_types}}); } elsif ($tpe->{code} =~ /\s+([^\s]+)\s+AS\s+TABLE\s+OF\s+([^;]+);/is) { %types_def = $self->_get_custom_types("varname $2"); push(@{$user_type{pg_types}} , \@{$types_def{pg_types}}); push(@{$user_type{src_types}}, \@{$types_def{src_types}}); } else { $self->logit("\tCan not found subtype for $tpe->{name} into code: $tpe->{code}\n", 1); } } } } return %user_type; } sub _extract_data { my ($self, $query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $proc) = @_; $0 = "ora2pg - querying table $table"; # Overwrite the query if REPLACE_QUERY is defined for this table if ($self->{replace_query}{"\L$table\E"}) { $query = $self->{replace_query}{"\L$table\E"}; if (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) { my $colpk = $self->{defined_pk}{"\L$table\E"}; if ($self->{preserve_case}) { $colpk = '"' . $colpk . '"'; } my $cond = " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; $cond = " ABS($colpk % $self->{oracle_copies}) = ?" if ($self->{is_mssql}); if ($query !~ s/\bWHERE\s+/WHERE $cond AND /) { if ($query !~ s/\b(ORDER\s+BY\s+.*)/WHERE $cond $1/) { $query .= " WHERE $cond"; } } } } my %user_type = (); my $rname = $part_name || $table; my $dbh = 0; my $sth = 0; my @has_custom_type = (); @{ $self->{data_cols}{$table} } = (); # Look for user defined type if (!$self->{is_mysql}) { for (my $idx = 0; $idx < scalar(@$stt); $idx++) { my $data_type = uc($stt->[$idx]) || ''; $data_type =~ s/\(.*//; # remove any precision # in case of user defined type try to gather the underlying base types if (!exists $self->{data_type}{$data_type} && exists $self->{user_type}{$stt->[$idx]}) { push(@has_custom_type, $idx); %{ $user_type{$idx} } = %{ $self->{user_type}{$stt->[$idx]} }; } } } if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { $self->logit("DEBUG: cloning Oracle database connection.\n", 1); $dbh = $self->{dbh}->clone(); # Force execution of initial command $self->_ora_initial_command($dbh); if (!$self->{is_mysql} && !$self->{is_mssql}) { # Force numeric format into the cloned session $self->_numeric_format($dbh); # Force datetime format into the cloned session $self->_datetime_format($dbh); # Set the action name on Oracle side to see which table is exported $dbh->do("CALL DBMS_APPLICATION_INFO.set_action('$table')") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } # Set row cache size $dbh->{RowCacheSize} = int($self->{data_limit}/10); if (exists $self->{local_data_limit}{$table}) { $dbh->{RowCacheSize} = $self->{local_data_limit}{$table}; } # prepare the query before execution if ($self->{is_mysql}) { $query =~ s/^SELECT\s+/SELECT \/\*\!40001 SQL_NO_CACHE \*\/ /s; $sth = $dbh->prepare($query, { mysql_use_result => 1, mysql_use_row => 1 }) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } elsif ($self->{is_mssql}) { $sth = $dbh->prepare($query) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->{'LongReadLen'} = $self->{longreadlen}; } else { if (!$self->{use_lob_locator}) { $sth = $dbh->prepare($query,{ora_piece_lob => 1, ora_piece_size => $self->{longreadlen}, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } else { $sth = $dbh->prepare($query,{'ora_auto_lob' => 0, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } } } else { # Set row cache size $self->{dbh}->{RowCacheSize} = int($self->{data_limit}/10); if (exists $self->{local_data_limit}{$table}) { $self->{dbh}->{RowCacheSize} = $self->{local_data_limit}{$table}; } # prepare the query before execution if ($self->{is_mysql}) { $query =~ s/^SELECT\s+/SELECT \/\*\!40001 SQL_NO_CACHE \*\/ /s; $sth = $self->{dbh}->prepare($query, { mysql_use_result => 1, mysql_use_row => 1 }) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } elsif ($self->{is_mssql}) { $sth = $self->{dbh}->prepare($query) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->{'LongReadLen'} = $self->{longreadlen}; } else { # Set the action name on Oracle side to see which table is exported $self->{dbh}->do("CALL DBMS_APPLICATION_INFO.set_action('$table')") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); if (!$self->{use_lob_locator}) { $sth = $self->{dbh}->prepare($query,{ora_piece_lob => 1, ora_piece_size => $self->{longreadlen}, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}); } else { $sth = $self->{dbh}->prepare($query,{'ora_auto_lob' => 0, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}); } if ($self->{dbh}->errstr =~ /ORA-00942/) { $self->logit("WARNING: table $table is not yet physically created and has no data.\n", 0, 0); # Only useful for single process return 0; } elsif ($self->{dbh}->errstr) { $self->logit("FATAL: _extract_data() " . $self->{dbh}->errstr . "\n", 1, 1); } } } # Extract data now by chunk of DATA_LIMIT and send them to a dedicated job $self->logit("Fetching all data from $rname tuples...\n", 1); my $start_time = time(); my $total_record = 0; my $total_row = $self->{tables}{$table}{table_info}{num_rows}; # Send current table in progress if (defined $pipe) { my $t_name = $part_name || $table; if ($proc ne '') { $pipe->print("TABLE EXPORT IN PROGESS: $t_name-part-$proc, start: $start_time, rows $total_row\n"); } else { $pipe->print("TABLE EXPORT IN PROGESS: $t_name, start: $start_time, rows $total_row\n"); } } my @params = (); if (defined $proc) { unshift(@params, $proc); $self->logit("Parallelizing on core #$proc with query: $query\n", 1); } if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { $sth->execute(@params) or $self->logit("FATAL: " . $dbh->errstr . ", SQL: $query\n", 0, 1); } else { $sth->execute(@params) or $self->logit("FATAL: " . $self->{dbh}->errstr . ", SQL: $query\n", 0, 1); } my $col_cond = $self->hs_cond($tt,$stt, $table); # Oracle allow direct retreiving of bchunk of data if (!$self->{is_mysql}) { my $data_limit = $self->{data_limit}; if (exists $self->{local_data_limit}{$table}) { $data_limit = $self->{local_data_limit}{$table}; } my $has_blob = 0; $has_blob = 1 if (grep(/LOB|XMLTYPE/, @$stt)); # With rows that not have custom type nor blob unless the user doesn't want to use lob locator if (($#has_custom_type == -1) && (!$has_blob || !$self->{use_lob_locator})) { while ( my $rows = $sth->fetchall_arrayref(undef,$data_limit)) { if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { if ($dbh->errstr) { $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0); last; } } elsif ( $self->{dbh}->errstr ) { $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); last; } $total_record += @$rows; $self->{current_total_row} += @$rows; $self->logit("DEBUG: number of rows $total_record extracted from table $table\n", 1); # Do we just want to test Oracle output speed if ($self->{oracle_speed} && !$self->{ora2pg_speed}) { my $tt_record = @$rows; $self->print_to_progressbar($table, $part_name, $procnum, $start_time, $total_record, $tt_record, $self->{tables}{$table}{table_info}{num_rows}); next; } if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { while ($self->{child_count} >= $self->{jobs}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->_dump_to_pg($proc, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); }; $self->{child_count}++; } else { $self->_dump_to_pg($proc, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); } } } else { my @rows = (); while ( my @row = $sth->fetchrow_array()) { if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { if ($dbh->errstr) { $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0); last; } } elsif ( $self->{dbh}->errstr ) { $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); last; } # Then foreach row use the returned lob locator to retrieve data # and all column with a LOB data type, extract data by chunk for (my $j = 0; $j <= $#$stt; $j++) { # Look for data based on custom type to replace the reference by the value if ($row[$j] =~ /^(?!(?!)\x{100})ARRAY\(0x/ && $stt->[$j] !~ /SDO_GEOMETRY/i) { my $data_type = uc($stt->[$j]) || ''; $data_type =~ s/\(.*//; # remove any precision $row[$j] = $self->set_custom_type_value($data_type, $user_type{$j}, $row[$j], $tt->[$j], 0); } # Retrieve LOB data from locator elsif (($stt->[$j] =~ /LOB|XMLTYPE/) && $row[$j]) { my $lob_content = ''; my $offset = 1; # Offsets start at 1, not 0 if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { # Get chunk size my $chunk_size = $self->{lob_chunk_size} || $dbh->ora_lob_chunk_size($row[$j]) || 8192; while (1) { my $lobdata = $dbh->ora_lob_read($row[$j], $offset, $chunk_size ); if ($dbh->errstr) { $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0) if ($dbh->errstr !~ /ORA-22831/); last; } last unless (defined $lobdata && length $lobdata); $offset += $chunk_size; $lob_content .= $lobdata; } } else { # Get chunk size my $chunk_size = $self->{lob_chunk_size} || $self->{dbh}->ora_lob_chunk_size($row[$j]) || 8192; while (1) { my $lobdata = $self->{dbh}->ora_lob_read($row[$j], $offset, $chunk_size ); if ($self->{dbh}->errstr) { $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0) if ($self->{dbh}->errstr !~ /ORA-22831/); last; } last unless (defined $lobdata && length $lobdata); $offset += $chunk_size; $lob_content .= $lobdata; } } if ($lob_content ne '') { $row[$j] = $lob_content; } else { $row[$j] = undef; } } elsif (($stt->[$j] =~ /LOB/) && !$row[$j]) { # This might handle case where the LOB is NULL and might prevent error: # DBD::Oracle::db::ora_lob_read: locator is not of type OCILobLocatorPtr $row[$j] = undef; } } $total_record++; $self->{current_total_row}++; push(@rows, [ @row ] ); if ($#rows == $data_limit) { # Do we just want to test Oracle output speed if ($self->{oracle_speed} && !$self->{ora2pg_speed}) { my $tt_record = @$rows; $self->print_to_progressbar($table, $part_name, $procnum, $start_time, $total_record, $tt_record, $self->{tables}{$table}{table_info}{num_rows}); next; } if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { while ($self->{child_count} >= $self->{jobs}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); }; $self->{child_count}++; } else { $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); } @rows = (); } } # Do we just want to test Oracle output speed # Do we just want to test Oracle output speed if ($self->{oracle_speed} && !$self->{ora2pg_speed}) { my $tt_record = @$rows; $self->print_to_progressbar($table, $part_name, $procnum, $start_time, $total_record, $tt_record, $self->{tables}{$table}{table_info}{num_rows}); next; } # Flush last extracted data if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { while ($self->{child_count} >= $self->{jobs}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); }; $self->{child_count}++; } else { $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); } @rows = (); } } else { my @rows = (); my $num_row = 0; while (my @row = $sth->fetchrow()) { push(@rows, \@row); $num_row++; if ($num_row == $self->{data_limit}) { $num_row = 0; $total_record += @rows; $self->{current_total_row} += @rows; # Do we just want to test Oracle output speed next if ($self->{oracle_speed} && !$self->{ora2pg_speed}); $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); @rows = (); } } if (!$self->{oracle_speed} || $self->{ora2pg_speed}) { $total_record += @rows; $self->{current_total_row} += @rows; $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); } } $sth->finish(); # When copy freeze is required, close the transaction if ($self->{copy_freeze} && !$self->{pg_dsn}) { if ($self->{file_per_table}) { $self->data_dump("COMMIT;\n", $table); } else { $self->dump("\nCOMMIT;\n"); } } # Close global data file in use when parallel table is used without output mutliprocess $self->close_export_file($self->{cfhout}) if (defined $self->{cfhout}); $self->{cfhout} = undef; if (!$self->{quiet} && !$self->{debug}) { if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1)) { my $end_time = time(); my $dt = $end_time - $self->{global_start_time}; my $rps = int($self->{current_total_row} / ($dt||1)); print STDERR "\n"; print STDERR $self->progress_bar($self->{current_total_row}, $self->{global_rows}, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec).") . "\n"; } } # Wait for all child end while ($self->{child_count} > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } if (defined $pipe) { my $t_name = $part_name || $table; my $t_time = time(); if ($proc ne '') { $pipe->print("TABLE EXPORT ENDED: $t_name-part-$proc, end: $t_time, rows $total_record\n"); } else { $pipe->print("TABLE EXPORT ENDED: $t_name, end: $t_time, rows $total_record\n"); } } $dbh->disconnect() if ($dbh); # Only useful for single process return $total_record; } sub log_error_copy { my ($self, $table, $s_out, $rows) = @_; my $outfile = ''; if ($self->{output_dir} && !$noprefix) { $outfile = $self->{output_dir} . '/'; } $outfile .= $table . '_error.log'; my $filehdl = new IO::File; $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't write to $outfile: $!\n", 0, 1); $filehdl->print($s_out); foreach my $row (@$rows) { $filehdl->print(join("\t", @$row) . "\n"); } $filehdl->print("\\.\n"); $self->close_export_file($filehdl); } sub log_error_insert { my ($self, $table, $sql_out) = @_; my $outfile = ''; if ($self->{output_dir} && !$noprefix) { $outfile = $self->{output_dir} . '/'; } $outfile .= $table . '_error.log'; my $filehdl = new IO::File; $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't write to $outfile: $!\n", 0, 1); $filehdl->print("$sql_out\n"); $self->close_export_file($filehdl); } sub print_to_progressbar { my ($self, $table, $part_name, $procnum, $ora_start_time, $total_row, $tt_record, $glob_total_record) = @_; my $end_time = time(); $ora_start_time = $end_time if (!$ora_start_time); my $dt = $end_time - $ora_start_time; my $rps = int($glob_total_record / ($dt||1)); my $t_name = $part_name || $table; if (!$self->{quiet} && !$self->{debug}) { # Send current table in progress if (defined $pipe) { if ($procnum ne '') { $pipe->print("CHUNK $$ DUMPED: $t_name-part-$procnum, time: $end_time, rows $tt_record\n"); } else { $pipe->print("CHUNK $$ DUMPED: $t_name, time: $end_time, rows $tt_record\n"); } } else { print STDERR $self->progress_bar($glob_total_record, $total_row, 25, '=', 'rows', "Table $t_name ($rps recs/sec)"), "\r"; } } elsif ($self->{debug}) { $self->logit("Extracted records from table $t_name: total_records = $glob_total_record (avg: $rps recs/sec)\n", 1); } } sub _dump_to_pg { my ($self, $procnum, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $ora_start_time, $part_name, $glob_total_record, %user_type) = @_; my @tempfiles = (); if ($^O !~ /MSWin32|dos/i) { push(@tempfiles, [ tempfile('tmp_ora2pgXXXXXX', SUFFIX => '', DIR => $TMP_DIR, UNLINK => 1 ) ]); } # Oracle source table or partition my $rname = $part_name || $table; # Destination PostgreSQL table (direct import to partition is not allowed with native partitioning) my $dname = $table; $dname = $part_name if (!$self->{pg_supports_partition}); if ($self->{pg_dsn}) { $0 = "ora2pg - sending data from table $rname to table $dname"; } else { $0 = "ora2pg - writing to file data from table $rname to table $dname"; } # Connect to PostgreSQL if direct import is enabled my $dbhdest = undef; if ($self->{pg_dsn} && !$self->{oracle_speed}) { $dbhdest = $self->_send_to_pgdb(); $self->logit("Dumping data from table $rname into PostgreSQL table $dname...\n", 1); $self->logit("Setting client_encoding to $self->{client_encoding}...\n", 1); my $s = $dbhdest->do( "SET client_encoding TO '\U$self->{client_encoding}\E';") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); if (!$self->{synchronous_commit}) { $self->logit("Disabling synchronous commit when writing to PostgreSQL...\n", 1); $s = $dbhdest->do("SET synchronous_commit TO off") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } # Build header of the file my $h_towrite = ''; foreach my $cmd (@$cmd_head) { if ($self->{pg_dsn} && !$self->{oracle_speed}) { $self->logit("Executing pre command to PostgreSQL: $cmd\n", 1); my $s = $dbhdest->do("$cmd") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } else { $h_towrite .= "$cmd\n"; } } # Build footer of the file my $e_towrite = ''; foreach my $cmd (@$cmd_foot) { if ($self->{pg_dsn} && !$self->{oracle_speed}) { my $s = $dbhdest->do("$cmd") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } else { $e_towrite .= "$cmd\n"; } } # Preparing data for output if ( !$sprep && ($#{$rows} >= 0) ) { my $data_limit = $self->{data_limit}; if (exists $self->{local_data_limit}{$table}) { $data_limit = $self->{local_data_limit}{$table}; } my $len = @$rows; $self->logit("DEBUG: Formatting bulk of $data_limit data (real: $len rows) for PostgreSQL.\n", 1); $self->format_data($rows, $tt, $self->{type}, $stt, \%user_type, $table); } # Add COPY header to the output my $sql_out = $s_out; # Creating output my $data_limit = $self->{data_limit}; if (exists $self->{local_data_limit}{$table}) { $data_limit = $self->{local_data_limit}{$table}; } $self->logit("DEBUG: Creating output for $data_limit tuples\n", 1); if ($self->{type} eq 'COPY') { if ($self->{pg_dsn}) { $sql_out =~ s/;$//; if (!$self->{oracle_speed}) { $self->logit("DEBUG: Sending COPY bulk output directly to PostgreSQL backend\n", 1); $dbhdest->do($sql_out) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); $sql_out = ''; my $skip_end = 0; foreach my $row (@$rows) { unless($dbhdest->pg_putcopydata(join("\t", @$row) . "\n")) { if ($self->{log_on_error}) { $self->logit("ERROR (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); $self->log_error_copy($table, $s_out, $rows); $skip_end = 1; last; } else { $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } } unless ($dbhdest->pg_putcopyend()) { if ($self->{log_on_error}) { $self->logit("ERROR (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); $self->log_error_copy($table, $s_out, $rows) if (!$skip_end); } else { $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } } else { foreach my $row (@$rows) { # do nothing, just add loop time nothing must be sent to PG } } } else { # then add data to the output map { $sql_out .= join("\t", @$_) . "\n"; } @$rows; $sql_out .= "\\.\n"; } } elsif (!$sprep) { $sql_out = ''; foreach my $row (@$rows) { $sql_out .= $s_out; $sql_out .= join(',', @$row) . ")"; if ($self->{insert_on_conflict}) { $sql_out .= " ON CONFLICT DO NOTHING"; } $sql_out .= ";\n"; } } # Insert data if we are in online processing mode if ($self->{pg_dsn}) { if ($self->{type} ne 'COPY') { if (!$sprep && !$self->{oracle_speed}) { $self->logit("DEBUG: Sending INSERT output directly to PostgreSQL backend\n", 1); unless($dbhdest->do("BEGIN;\n" . $sql_out . "COMMIT;\n")) { if ($self->{log_on_error}) { $self->logit("WARNING (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); $self->log_error_insert($table, "BEGIN;\n" . $sql_out . "COMMIT;\n"); } else { $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } } else { my $ps = undef; if (!$self->{oracle_speed}) { $ps = $dbhdest->prepare($sprep) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } my @date_cols = (); my @bool_cols = (); for (my $i = 0; $i <= $#{$tt}; $i++) { if ($tt->[$i] eq 'bytea') { if (!$self->{oracle_speed}) { $ps->bind_param($i+1, undef, { pg_type => DBD::Pg::PG_BYTEA }); } } elsif ($tt->[$i] eq 'boolean') { push(@bool_cols, $i); } elsif ($tt->[$i] =~ /(date|time)/i) { push(@date_cols, $i); } } $self->logit("DEBUG: Sending INSERT bulk output directly to PostgreSQL backend\n", 1); my $col_cond = $self->hs_cond($tt, $stt, $table); foreach my $row (@$rows) { # Even with prepared statement we need to replace zero date foreach my $j (@date_cols) { if ($row->[$j] =~ /^0000-/) { if (!$self->{replace_zero_date}) { $row->[$j] = undef; } else { $row->[$j] = $self->{replace_zero_date}; } } } # Format user defined type and geometry data $self->format_data_row($row,$tt,'INSERT', $stt, \%user_type, $table, $col_cond, 1); # Replace boolean 't' and 'f' by 0 and 1 for bind parameters. foreach my $j (@bool_cols) { ($row->[$j] eq "'f'") ? $row->[$j] = 0 : $row->[$j] = 1; } # Apply bind parmeters if (!$self->{oracle_speed}) { unless ( $ps->execute(@$row) ) { if ($self->{log_on_error}) { $self->logit("ERROR (log error enabled): " . $ps->errstr . "\n", 0, 0); $s_out =~ s/\([,\?]+\)/\(/; $self->format_data_row($row,$tt,'INSERT', $stt, \%user_type, $table, $col_cond); $self->log_error_insert($table, $s_out . join(',', @$row) . ");\n"); } else { $self->logit("FATAL: " . $ps->errstr . "\n", 0, 1); } } } } if (!$self->{oracle_speed}) { $ps->finish(); } } } } else { if ($part_name && $self->{rename_partition}) { $part_name = $table . '_' . $part_name; } $sql_out = $h_towrite . $sql_out . $e_towrite; if (!$self->{oracle_speed}) { $self->data_dump($sql_out, $table, $part_name); } } my $total_row = $self->{tables}{$table}{table_info}{num_rows}; my $tt_record = @$rows; $dbhdest->disconnect() if ($dbhdest); $self->print_to_progressbar($table, $part_name, $procnum, $ora_start_time, $total_row, $tt_record, $glob_total_record); if ($^O !~ /MSWin32|dos/i) { if (defined $tempfiles[0]->[0]) { close($tempfiles[0]->[0]); } unlink($tempfiles[0]->[1]) if (-e $tempfiles[0]->[1]); } } sub _pload_to_pg { my ($self, $idx, $query, @settings) = @_; if (!$self->{pg_dsn}) { $self->logit("FATAL: No connection to PostgreSQL database set, aborting...\n", 0, 1); } my @tempfiles = (); if ($^O !~ /MSWin32|dos/i) { push(@tempfiles, [ tempfile('tmp_ora2pgXXXXXX', SUFFIX => '', DIR => $TMP_DIR, UNLINK => 1 ) ]); } # Open a connection to the postgreSQL database $0 = "ora2pg - sending query to PostgreSQL database"; # Connect to PostgreSQL if direct import is enabled my $dbhdest = $self->_send_to_pgdb(); $self->logit("Loading query #$idx: $query\n", 1); if ($#settings == -1) { $self->logit("Applying settings from configuration\n", 1); # Apply setting from configuration $dbhdest->do( "SET client_encoding TO '\U$self->{client_encoding}\E';") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); my $search_path = $self->set_search_path(); if ($search_path) { $self->logit("Setting search_path using: $search_path...\n", 1); $dbhdest->do($search_path) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } else { $self->logit("Applying settings from input file\n", 1); # Apply setting from source file foreach my $set (@settings) { $dbhdest->do($set) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); } } # Execute query $dbhdest->do("$query") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); $dbhdest->disconnect() if ($dbhdest); if ($^O !~ /MSWin32|dos/i) { if (defined $tempfiles[0]->[0]) { close($tempfiles[0]->[0]); } unlink($tempfiles[0]->[1]) if (-e $tempfiles[0]->[1]); } } # Global array, to store the converted values my @bytea_array; sub build_escape_bytea { foreach my $tmp (0..255) { my $out; if ($tmp >= 32 and $tmp <= 126) { if ($tmp == 92) { $out = '\\\\134'; } elsif ($tmp == 39) { $out = '\\\\047'; } else { $out = chr($tmp); } } else { $out = sprintf('\\\\%03o',$tmp); } $bytea_array[$tmp] = $out; } } =head2 escape_bytea This function return an escaped bytea entry for Pg. =cut sub escape_bytea { my $data = shift; # In this function, we use the array built by build_escape_bytea my @array= unpack("C*", $data); foreach my $elt (@array) { $elt = $bytea_array[$elt]; } return join('', @array); } =head2 _show_infos This function display a list of schema, table or column only to stdout. =cut sub _show_infos { my ($self, $type) = @_; if ($type eq 'SHOW_ENCODING') { my ($db_encoding, $collation, $client_encoding, $timestamp_format, $date_format) = $self->_get_encoding($self->{dbh}); if ($self->{is_mysql}) { $self->logit("Current encoding settings that will be used by Ora2Pg:\n", 0); $self->logit("\tMySQL database and client encoding: $self->{nls_lang}\n", 0); $self->logit("\tMySQL collation encoding: $self->{nls_nchar}\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING $self->{client_encoding}\n", 0); $self->logit("\tPerl output encoding '$self->{binmode}'\n", 0); $self->logit("Showing current MySQL encoding and possible PostgreSQL client encoding:\n", 0); $self->logit("\tMySQL database and client encoding: $db_encoding\n", 0); $self->logit("\tMySQL collation encoding: $collation\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING: $client_encoding\n", 0); $self->logit("MySQL SQL mode: $self->{mysql_mode}\n", 0); } elsif ($self->{is_mssql}) { $self->logit("Current encoding settings that will be used by Ora2Pg:\n", 0); $self->logit("\tMSSQL database and client encoding: utf8\n", 0); $self->logit("\tMSSQL collation encoding: $self->{nls_nchar}\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING: $self->{client_encoding}\n", 0); $self->logit("\tPerl output encoding '$self->{binmode}'\n", 0); $self->logit("\tOra2Pg use UTF8 export to export from MSSQL, change to NSL_LANG and\n", 0); $self->logit("\tNLS_NCHAR have no effect. CLIENT_ENCODING must be set to UFT8\n", 0); $self->logit("Showing current MSSQL encoding and possible PostgreSQL client encoding:\n", 0); $self->logit("\tMSSQL database encoding: $self->{nls_lang}\n", 0); $self->logit("\tMSSQL collation encoding: $self->{nls_nchar}\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING: $client_encoding\n", 0); } else { $self->logit("Current encoding settings that will be used by Ora2Pg:\n", 0); $self->logit("\tOracle NLS_LANG $self->{nls_lang}\n", 0); $self->logit("\tOracle NLS_NCHAR $self->{nls_nchar}\n", 0); if ($self->{enable_microsecond}) { $self->logit("\tOracle NLS_TIMESTAMP_FORMAT YYYY-MM-DD HH24:MI:SS.FF\n", 0); } else { $self->logit("\tOracle NLS_TIMESTAMP_FORMAT YYYY-MM-DD HH24:MI:SS\n", 0); } $self->logit("\tOracle NLS_DATE_FORMAT YYYY-MM-DD HH24:MI:SS\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING $self->{client_encoding}\n", 0); $self->logit("\tPerl output encoding '$self->{binmode}'\n", 0); $self->logit("Showing current Oracle encoding and possible PostgreSQL client encoding:\n", 0); $self->logit("\tOracle NLS_LANG $db_encoding\n", 0); $self->logit("\tOracle NLS_NCHAR $collation\n", 0); $self->logit("\tOracle NLS_TIMESTAMP_FORMAT $timestamp_format\n", 0); $self->logit("\tOracle NLS_DATE_FORMAT $date_format\n", 0); $self->logit("\tPostgreSQL CLIENT_ENCODING $client_encoding\n", 0); } } elsif ($type eq 'SHOW_VERSION') { $self->logit("Showing Database Version...\n", 1); $self->logit("$self->{db_version}\n", 0); } elsif ($type eq 'SHOW_REPORT') { # Get all tables information specified by the DBI method table_info if ($self->{is_mssql}) { my $sth = $self->_schema_list() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$self->{local_schemas}}, $row[0]); } $sth->finish(); } if ($#{$self->{local_schemas}} >= 0) { $self->{local_schemas_regex} = '(' . join('|', @{$self->{local_schemas}}) . ')'; } print STDERR "Reporting Oracle Content...\n" if ($self->{debug}); my $uncovered_score = 'Ora2Pg::PLSQL::UNCOVERED_SCORE'; if ($self->{is_mysql}) { $uncovered_score = 'Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE'; } elsif ($self->{is_mssql}) { $uncovered_score = 'Ora2Pg::PLSQL::UNCOVERED_MSSQL_SCORE'; } # Get Oracle database version and size print STDERR "Looking at Oracle server version...\n" if ($self->{debug}); my $ver = $self->_get_version(); print STDERR "Looking at Oracle database size...\n" if ($self->{debug}); my $size = $self->_get_database_size(); # Get the list of all database objects print STDERR "Looking at Oracle defined objects...\n" if ($self->{debug}); my %objects = $self->_get_objects(); # Extract all tables informations my %all_indexes = (); $self->{skip_fkeys} = $self->{skip_indices} = $self->{skip_indexes} = $self->{skip_checks} = 0; $self->{view_as_table} = (); $self->{mview_as_table} = (); print STDERR "Looking at table definition...\n" if ($self->{debug}); $self->_tables(1); my $total_index = 0; my $total_table_objects = 0; my $total_index_objects = 0; foreach my $table (sort keys %{$self->{tables}}) { $total_table_objects++; push(@exported_indexes, $self->_exportable_indexes($table, %{$self->{tables}{$table}{indexes}})); $total_index_objects += scalar keys %{$self->{tables}{$table}{indexes}}; foreach my $idx (sort keys %{$self->{tables}{$table}{idx_type}}) { next if (!grep(/^$idx$/i, @exported_indexes)); my $typ = $self->{tables}{$table}{idx_type}{$idx}{type}; push(@{$all_indexes{$typ}}, $idx); $total_index++; } } # Convert Oracle user defined type to PostgreSQL if (!$self->{is_mysql}) { $self->_types(); foreach my $tpe (sort { $a->{pos} <=> $b->{pos} } @{$self->{types}}) { # We dont want the result but only the array @{$self->{types}} # define in the _convert_type() function $self->_convert_type($tpe->{code}, $tpe->{owner}); } } print STDERR "Looking at views definition...\n" if ($self->{debug}); my %view_infos = (); %view_infos = $self->_get_views() if ($self->{estimate_cost}); # Get definition of Database Link print STDERR "Looking at database links...\n" if ($self->{debug}); my %dblink = $self->_get_dblink(); $objects{'DATABASE LINK'} = scalar keys %dblink; print STDERR "\tFound $objects{'DATABASE LINK'} DATABASE LINK.\n" if ($self->{debug}); # Get Jobs print STDERR "Looking at jobs...\n" if ($self->{debug}); my %jobs = $self->_get_job(); $objects{'JOB'} = scalar keys %jobs; print STDERR "\tFound $objects{'JOB'} JOB.\n" if ($self->{debug}); # Get synonym information print STDERR "Looking at synonyms...\n" if ($self->{debug}); my %synonyms = $self->_synonyms(); $objects{'SYNONYM'} = scalar keys %synonyms; print STDERR "\tFound $objects{'SYNONYM'} SYNONYM.\n" if ($self->{debug}); # Get all global temporary tables print STDERR "Looking at global temporary table...\n" if ($self->{debug}); my %global_tables = $self->_global_temp_table_info(); $objects{'GLOBAL TEMPORARY TABLE'} = scalar keys %global_tables; print STDERR "\tFound $objects{'GLOBAL TEMPORARY TABLE'} GLOBAL TEMPORARY TABLE.\n" if ($self->{debug}); # Look for encrypted columns and identity columns my %encrypted_column = (); if ($self->{db_version} !~ /Release [89]/) { print STDERR "Looking at encrypted columns...\n" if ($self->{debug}); %encrypted_column = $self->_encrypted_columns('',$self->{schema}); print STDERR "\tFound ", scalar keys %encrypted_column, " encrypted column.\n" if ($self->{debug}); print STDERR "Looking at identity columns...\n" if ($self->{debug}); # Identity column are collected in call to sub _tables() above print STDERR "\tFound ", scalar keys %{$self->{identity_info}}, " identity column.\n" if ($self->{debug}); } # Look at all database objects to compute report my %report_info = (); $report_info{'Version'} = $ver || 'Unknown'; $report_info{'Schema'} = $self->{schema} || ''; $report_info{'Size'} = $size || 'Unknown'; my $idx = 0; my $num_total_obj = scalar keys %objects; foreach my $typ (sort keys %objects) { $idx++; next if ($typ eq 'EVALUATION CONTEXT'); # Do not care about rule evaluation context next if ($self->{is_mysql} && $typ eq 'SYNONYM'); next if ($typ eq 'PACKAGE'); # Package are scanned with PACKAGE BODY not PACKAGE objects print STDERR "Building report for object $typ...\n" if ($self->{debug}); if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($idx, $num_total_obj, 25, '=', 'objects types', "inspecting object $typ" ), "\r"; } $report_info{'Objects'}{$typ}{'number'} = 0; $report_info{'Objects'}{$typ}{'invalid'} = 0; if (!grep(/^$typ$/, 'DATABASE LINK', 'JOB', 'TABLE', 'INDEX', 'SYNONYM','GLOBAL TEMPORARY TABLE')) { for (my $i = 0; $i <= $#{$objects{$typ}}; $i++) { $report_info{'Objects'}{$typ}{'number'}++; $report_info{'Objects'}{$typ}{'invalid'}++ if ($objects{$typ}[$i]->{invalid}); } } elsif ($typ eq 'TABLE') { $report_info{'Objects'}{$typ}{'number'} = $total_table_objects; } elsif ($typ eq 'INDEX') { $report_info{'Objects'}{$typ}{'number'} = $total_index_objects; } else { $report_info{'Objects'}{$typ}{'number'} = $objects{$typ}; } $report_info{'total_object_invalid'} += $report_info{'Objects'}{$typ}{'invalid'}; $report_info{'total_object_number'} += $report_info{'Objects'}{$typ}{'number'}; if ($report_info{'Objects'}{$typ}{'number'} > 0) { $report_info{'Objects'}{$typ}{'real_number'} = ($report_info{'Objects'}{$typ}{'number'} - $report_info{'Objects'}{$typ}{'invalid'}); $report_info{'Objects'}{$typ}{'real_number'} = $report_info{'Objects'}{$typ}{'number'} if ($self->{export_invalid}); } if ($self->{estimate_cost}) { $report_info{'Objects'}{$typ}{'cost_value'} = ($report_info{'Objects'}{$typ}{'real_number'}*$Ora2Pg::PLSQL::OBJECT_SCORE{$typ}); # Minimal unit is 1 $report_info{'Objects'}{$typ}{'cost_value'} = 1 if ($report_info{'Objects'}{$typ}{'cost_value'} =~ /^0\./); # For some object's type do not set migration unit upper than 2 days. if (grep(/^$typ$/, 'TABLE PARTITION', 'GLOBAL TEMPORARY TABLE', 'TRIGGER', 'VIEW')) { $report_info{'Objects'}{$typ}{'cost_value'} = 168 if ($report_info{'Objects'}{$typ}{'cost_value'} > 168); if (grep(/^$typ$/, 'TRIGGER', 'VIEW') && $report_info{'Objects'}{$typ}{'real_number'} > 500) { $report_info{'Objects'}{$typ}{'cost_value'} += 84 * int(($report_info{'Objects'}{$typ}{'real_number'} - 500) / 500); } } elsif (grep(/^$typ$/, 'TABLE', 'INDEX', 'SYNONYM')) { $report_info{'Objects'}{$typ}{'cost_value'} = 84 if ($report_info{'Objects'}{$typ}{'cost_value'} > 84); } } if ($typ eq 'INDEX') { my $bitmap = 0; foreach my $t (sort keys %INDEX_TYPE) { my $len = ($#{$all_indexes{$t}}+1); $report_info{'Objects'}{$typ}{'detail'} .= "\L$len $INDEX_TYPE{$t} index(es)\E\n" if ($len); if ($self->{estimate_cost} && $len && ( ($t =~ /FUNCTION.*NORMAL/) || ($t eq 'FUNCTION-BASED BITMAP') ) ) { $report_info{'Objects'}{$typ}{'cost_value'} += ($len * $Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION-BASED-INDEX'}); } if ($self->{estimate_cost} && $len && ($t =~ /REV/)) { $report_info{'Objects'}{$typ}{'cost_value'} += ($len * $Ora2Pg::PLSQL::OBJECT_SCORE{'REV-INDEX'}); } } $report_info{'Objects'}{$typ}{'cost_value'} += ($Ora2Pg::PLSQL::OBJECT_SCORE{$typ}*$total_index) if ($self->{estimate_cost}); $report_info{'Objects'}{$typ}{'comment'} = "$total_index index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL."; my $hash_index = ''; if ($self->{pg_version} < 10) { $hash_index = ' and hash index(es) will be exported as b-tree index(es) if any'; } if (!$self->{is_mysql}) { my $bitmap = 'Bitmap'; if ($self->{bitmap_as_gin}) { $bitmap = 'Bitmap will be exported as btree_gin index(es)'; } $report_info{'Objects'}{$typ}{'comment'} .= " $bitmap$hash_index. Domain index are exported as b-tree but commented to be edited to mainly use FTS. Cluster, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. Use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns."; } else { $report_info{'Objects'}{$typ}{'comment'} .= "$hash_index. Use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. Fulltext search indexes will be replaced by using a dedicated tsvector column, Ora2Pg will set the DDL to create the column, function and trigger together with the index."; } } elsif ($typ eq 'MATERIALIZED VIEW') { $report_info{'Objects'}{$typ}{'comment'}= "All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed."; my %mview_infos = $self->_get_materialized_views(); my $oncommit = 0; foreach my $mview (sort keys %mview_infos) { if ($mview_infos{$mview}{refresh_mode} eq 'COMMIT') { $oncommit++; $report_info{'Objects'}{$typ}{'detail'} .= "$mview, "; } } if ($oncommit) { $report_info{'Objects'}{$typ}{'detail'} =~ s/, $//; $report_info{'Objects'}{$typ}{'detail'} = "$oncommit materialized views are refreshed on commit ($report_info{'Objects'}{$typ}{'detail'}), this is not supported by PostgreSQL, you will need to use triggers to have the same behavior or use a simple view."; } } elsif ($typ eq 'TABLE') { my $exttb = scalar keys %{$self->{external_table}}; if ($exttb) { if (!$self->{external_to_fdw}) { $report_info{'Objects'}{$typ}{'comment'} = "$exttb external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files."; } else { $report_info{'Objects'}{$typ}{'comment'} = "$exttb external table(s) will be exported as file_fdw foreign table. See EXTERNAL_TO_FDW configuration directive to export as standard table or use COPY in your code if you just want to load data from external files."; } } my %table_detail = (); my $virt_column = 0; my @done = (); my $id = 0; my $total_check = 0; my $total_row_num = 0; my $rdbms_error_table = 0; my $quartz_scheduler = 0; my @maxtablen = (); my @maxcollen = (); # Set the table information for each class found foreach my $t (sort keys %{$self->{tables}}) { my $tbname = $t; $tbname =~ s/.*\.//; # Set the total number of rows $total_row_num += $self->{tables}{$t}{table_info}{num_rows}; # Look if this is a RDBMS table $rdbms_error_table++ if ($tbname =~ /^ERR\$_/i); # Object name too long push(@maxtablen, $t) if (length($tbname) > 63); # Look if this is a Quartz Scheduler table $quartz_scheduler++ if ($tbname =~ /^QRTZ_/i); # Look at reserved words if tablename is found my $r = $self->is_reserved_words($t); if (($r > 0) && ($r != 3)) { $table_detail{'reserved words in table name'}++; $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough } # Get fields informations foreach my $k (sort {$self->{tables}{$t}{column_info}{$a}[11] <=> $self->{tables}{$t}{column_info}{$a}[11]} keys %{$self->{tables}{$t}{column_info}}) { # Column name too long push(@maxcollen, "$t.$self->{tables}{$t}{column_info}{$k}[0]") if (length($self->{tables}{$t}{column_info}{$k}[0]) > 63); $r = $self->is_reserved_words($self->{tables}{$t}{column_info}{$k}[0]); if (($r > 0) && ($r != 3)) { $table_detail{'reserved words in column name'}++; $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough } elsif ($r == 3) { $table_detail{'system columns in column name'}++; $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough } $self->{tables}{$t}{column_info}{$k}[1] =~ s/TIMESTAMP\(\d+\)/TIMESTAMP/i; if (!exists $self->{data_type}{uc($self->{tables}{$t}{column_info}{$k}[1])}) { $table_detail{'unknown types'}++; } if ( (uc($self->{tables}{$t}{column_info}{$k}[1]) eq 'NUMBER') && ($self->{tables}{$t}{column_info}{$k}[2] eq '') ) { $table_detail{'numbers with no precision'}++; } if ( $self->{data_type}{uc($self->{tables}{$t}{column_info}{$k}[1])} eq 'bytea' ) { $table_detail{'binary columns'}++; } } # Get check constraints information related to this table my $constraints = $self->_count_check_constraint($self->{tables}{$t}{check_constraint}); $total_check += $constraints; if ($self->{estimate_cost} && $constraints >= 0) { $report_info{'Objects'}{$typ}{'cost_value'} += $constraints * $Ora2Pg::PLSQL::OBJECT_SCORE{'CHECK'}; } } $report_info{'Objects'}{$typ}{'comment'} .= " $total_check check constraint(s)." if ($total_check); foreach my $d (sort keys %table_detail) { $report_info{'Objects'}{$typ}{'comment'} .= "\L$table_detail{$d} $d\E.\n"; } $report_info{'Objects'}{$typ}{'detail'} .= "Total number of rows: $total_row_num\n"; $report_info{'Objects'}{$typ}{'detail'} .= "Top $self->{top_max} of tables sorted by number of rows:\n"; my $j = 1; foreach my $t (sort {$self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows}} keys %{$self->{tables}}) { next if ($self->{tables}{$t}{table_info}{num_rows} == 0); $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E has $self->{tables}{$t}{table_info}{num_rows} rows\n"; $j++; last if ($j > $self->{top_max}); } $report_info{'Objects'}{$typ}{'detail'} .= "Top $self->{top_max} of largest tables:\n"; $j = 1; if ($self->{is_mssql} || $self->{is_mysql}) { foreach my $t (sort {$self->{tables}{$b}{table_info}{size} <=> $self->{tables}{$a}{table_info}{size}} keys %{$self->{tables}}) { next if ($self->{tables}{$t}{table_info}{size} == 0); $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E: $self->{tables}{$t}{table_info}{size} MB (" . $self->{tables}{$t}{table_info}{num_rows} . " rows)\n"; $j++; last if ($j > $self->{top_max}); } } else { # Because we avoid using JOIN when querying the Oracle catalog, look for all table size my %largest_table = (); %largest_table = $self->_get_largest_tables() if ($self->{is_mysql}); foreach my $t (sort { $largest_table{$b} <=> $largest_table{$a} } keys %largest_table) { $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E: $largest_table{$t} MB (" . $self->{tables}{$t}{table_info}{num_rows} . " rows)\n"; $j++; last if ($j > $self->{top_max}); } } $comment = "Nothing particular." if (!$comment); $report_info{'Objects'}{$typ}{'cost_value'} =~ s/(\.\d).*$/$1/; if (scalar keys %encrypted_column > 0) { $report_info{'Objects'}{$typ}{'comment'} .= "\n" . (scalar keys %encrypted_column) . " encrypted column(s).\n"; foreach my $k (sort keys %encrypted_column) { $report_info{'Objects'}{$typ}{'comment'} .= "\L$k\E ($encrypted_column{$k})\n"; } $report_info{'Objects'}{$typ}{'comment'} .= ". You must use the pg_crypto extension to use encryption.\n"; if ($self->{estimate_cost}) { $report_info{'Objects'}{$typ}{'cost_value'} += (scalar keys %encrypted_column) * $Ora2Pg::PLSQL::OBJECT_SCORE{'ENCRYPTED COLUMN'}; } $report_info{'Objects'}{$typ}{'comment'} .= "Table(s) name too long: " . ($#maxtablen+1) . "\n" if ($#maxtablen >= 0); $report_info{'Objects'}{$typ}{'detail'} .= "List of table(s) name too long:\n" . join(",\n", @maxtablen) . "\n" if ($#maxtablen >= 0); $report_info{'Objects'}{$typ}{'comment'} .= "Column(s) name too long: " . ($#maxcollen+1) . "\n" if ($#maxcollen >= 0); $report_info{'Objects'}{$typ}{'detail'} .= "List of column(s) name too long:\n" . join(",\n", @maxcollen) . "\n" if ($#maxcollen >= 0); $report_info{'Objects'}{$typ}{'comment'} .= "RDBMS_ERROR is used on $rdbms_error_table tables.\n" if ($rdbms_error_table); $report_info{'Objects'}{$typ}{'comment'} .= "Quartz Scheduler looks to be used.\n" if ($quartz_scheduler); } if (scalar keys %{$self->{identity_info}} > 0) { $report_info{'Objects'}{$typ}{'comment'} .= "\n" . (scalar keys %{$self->{identity_info}}) . " identity column(s).\n"; $report_info{'Objects'}{$typ}{'comment'} .= " Identity columns are fully supported since PG10.\n"; } } elsif ($typ eq 'TYPE') { my $total_type = $report_info{'Objects'}{'TYPE'}{'number'}; foreach my $t (sort keys %{$self->{type_of_type}}) { $total_type-- if (grep(/^$t$/, 'Associative Arrays','Type Boby','Type with member method', 'Type Ref Cursor')); $report_info{'Objects'}{$typ}{'detail'} .= "\L$self->{type_of_type}{$t} $t\E\n" if ($self->{type_of_type}{$t}); } $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{$typ}*$total_type) if ($self->{estimate_cost}); $report_info{'Objects'}{$typ}{'comment'} = "$total_type type(s) are concerned by the export, others are not supported. Note that Type inherited and Subtype are converted as table, type inheritance is not supported."; } elsif ($typ eq 'TYPE BODY') { $report_info{'Objects'}{$typ}{'comment'} = "Export of type with member method are not supported, they will not be exported."; } elsif ($typ eq 'TRIGGER') { my $triggers = $self->_get_triggers(); my $total_size = 0; foreach my $trig (@{$triggers}) { # Remove comment and text constant, they are not useful in assessment $self->_remove_comments(\$trig->[4]); $self->{comment_values} = (); $self->{text_values} = (); $self->{text_values_pos} = 0; if ($self->{is_mysql}) { $trig->[4] = $self->_convert_function($trig->[8], $trig->[4], $trig->[0]); } else { $trig->[4] = $self->_convert_function($trig->[8], $trig->[4]); } $total_size += length($trig->[4]); if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $trig->[4], 'TRIGGER'); $report_info{'Objects'}{$typ}{'cost_value'} += $cost; $report_info{'Objects'}{$typ}{'detail'} .= "\L$trig->[0]: $cost\E\n"; $report_info{full_trigger_details}{"\L$trig->[0]\E"}{count} = $cost; foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$d}); $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= "\t$d => $cost_detail{$d}"; $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= "\n"; push(@{$report_info{full_trigger_details}{"\L$trig->[0]\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); } } } $report_info{'Objects'}{$typ}{'comment'} = "Total size of trigger code: $total_size bytes."; } elsif ($typ eq 'SEQUENCE') { $report_info{'Objects'}{$typ}{'comment'} = "Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name')."; } elsif ($typ eq 'FUNCTION') { my $functions = $self->_get_functions(); my $total_size = 0; foreach my $fct (keys %{$functions}) { # Remove comment and text constant, they are not useful in assessment $self->_remove_comments(\$functions->{$fct}{text}); $self->{comment_values} = (); $self->{text_values} = (); $self->{text_values_pos} = 0; if ($self->{is_mysql}) { $functions->{$fct}{text} = $self->_convert_function($functions->{$fct}{owner}, $functions->{$fct}{text}, $fct); } else { $functions->{$fct}{text} = $self->_convert_function($functions->{$fct}{owner}, $functions->{$fct}{text}); } $total_size += length($functions->{$fct}{text}); if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $functions->{$fct}{text}, 'FUNCTION'); $report_info{'Objects'}{$typ}{'cost_value'} += $cost; $report_info{'Objects'}{$typ}{'detail'} .= "\L$fct: $cost\E\n"; $report_info{full_function_details}{"\L$fct\E"}{count} = $cost; foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$d}); $report_info{full_function_details}{"\L$fct\E"}{info} .= "\t$d => $cost_detail{$d}"; $report_info{full_function_details}{"\L$fct\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); $report_info{full_function_details}{"\L$fct\E"}{info} .= "\n"; push(@{$report_info{full_function_details}{"\L$fct\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); } } } $report_info{'Objects'}{$typ}{'comment'} = "Total size of function code: $total_size bytes."; } elsif ($typ eq 'PROCEDURE') { my $procedures = $self->_get_procedures(); my $total_size = 0; foreach my $proc (keys %{$procedures}) { # Remove comment and text constant, they are not useful in assessment $self->_remove_comments(\$procedures->{$proc}{text}); $self->{comment_values} = (); $self->{text_values} = (); $self->{text_values_pos} = 0; if ($self->{is_mysql}) { $procedures->{$proc}{text} = $self->_convert_function($procedures->{$proc}{owner}, $procedures->{$proc}{text}, $proc); } else { $procedures->{$proc}{text} = $self->_convert_function($procedures->{$proc}{owner}, $procedures->{$proc}{text}); } $total_size += length($procedures->{$proc}{text}); if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $procedures->{$proc}{text}, 'PROCEDURE'); $report_info{'Objects'}{$typ}{'cost_value'} += $cost; $report_info{'Objects'}{$typ}{'detail'} .= "\L$proc: $cost\E\n"; $report_info{full_procedure_details}{"\L$proc\E"}{count} = $cost; foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$d}); $report_info{full_procedure_details}{"\L$proc\E"}{info} .= "\t$d => $cost_detail{$d}"; $report_info{full_procedure_details}{"\L$proc\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); $report_info{full_procedure_details}{"\L$proc\E"}{info} .= "\n"; push(@{$report_info{full_procedure_details}{"\L$proc\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); } } } $report_info{'Objects'}{$typ}{'comment'} = "Total size of procedure code: $total_size bytes."; } elsif ($typ eq 'PACKAGE BODY') { $self->{packages} = $self->_get_packages(); my $total_size = 0; my $number_fct = 0; my $number_pkg = 0; foreach my $pkg (sort keys %{$self->{packages}}) { next if (!$self->{packages}{$pkg}{text}); $number_pkg++; $total_size += length($self->{packages}{$pkg}{text}); # Remove comment and text constant, they are not useful in assessment $self->_remove_comments(\$self->{packages}{$pkg}{text}); $self->{comment_values} = (); $self->{text_values} = (); $self->{text_values_pos} = 0; my @codes = split(/CREATE(?: OR REPLACE)?(?: EDITIONABLE| NONEDITIONABLE)? PACKAGE\s+/i, $self->{packages}{$pkg}{text}); foreach my $txt (@codes) { next if ($txt !~ /^BODY\s+/is); my %infos = $self->_lookup_package("CREATE OR REPLACE PACKAGE $txt"); foreach my $f (sort keys %infos) { next if (!$f); if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $infos{$f}{code}, $infos{$f}{type}); $report_info{'Objects'}{$typ}{'cost_value'} += $cost; $report_info{'Objects'}{$typ}{'detail'} .= "\L$f: $cost\E\n"; $report_info{full_package_details}{"\L$f\E"}{count} = $cost; foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$d}); $report_info{full_package_details}{"\L$f\E"}{info} .= "\t$d => $cost_detail{$d}"; $report_info{full_package_details}{"\L$f\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); $report_info{full_package_details}{"\L$f\E"}{info} .= "\n"; push(@{$report_info{full_package_details}{"\L$f\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); } } $number_fct++; } } } $self->{packages} = (); if ($self->{estimate_cost}) { $report_info{'Objects'}{$typ}{'cost_value'} += ($number_pkg*$Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}); } $report_info{'Objects'}{$typ}{'comment'} = "Total size of package code: $total_size bytes. Number of procedures and functions found inside those packages: $number_fct."; } elsif ( ($typ eq 'SYNONYM') && !$self->{is_mysql} ) { foreach my $t (sort {$a cmp $b} keys %synonyms) { if ($synonyms{$t}{dblink}) { $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E is a link to \L$synonyms{$t}{table_owner}.$synonyms{$t}{table_name}\@$synonyms{$t}{dblink}\E\n"; } else { $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E is an alias to $synonyms{$t}{table_owner}.$synonyms{$t}{table_name}\n"; } } $report_info{'Objects'}{$typ}{'comment'} = "SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround is to use views or set the PostgreSQL search_path in your session to access object outside the current schema."; } elsif ($typ eq 'INDEX PARTITION') { $report_info{'Objects'}{$typ}{'comment'} = "Only local indexes partition are exported, they are build on the column used for the partitioning."; } elsif ($typ eq 'TABLE PARTITION') { my %partitions = $self->_get_partitions_list(); foreach my $t (sort keys %partitions) { $report_info{'Objects'}{$typ}{'detail'} .= "$t\n"; } $report_info{'Objects'}{$typ}{'comment'} = "Partitions are well supported by PostgreSQL except key partition which will not be exported."; } elsif ($typ eq 'GLOBAL TEMPORARY TABLE') { $report_info{'Objects'}{$typ}{'comment'} = "Global temporary table are not supported by PostgreSQL and will not be exported. You will have to rewrite some application code to match the PostgreSQL temporary table behavior."; foreach my $t (sort keys %global_tables) { $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E\n"; } } elsif ($typ eq 'CLUSTER') { $report_info{'Objects'}{$typ}{'comment'} = "Clusters are not supported by PostgreSQL and will not be exported."; } elsif ($typ eq 'VIEW') { if ($self->{estimate_cost}) { foreach my $view (sort keys %view_infos) { # Remove unsupported definitions from the ddl statement $view_infos{$view}{text} =~ s/\s*WITH\s+READ\s+ONLY//is; $view_infos{$view}{text} =~ s/\s*OF\s+([^\s]+)\s+(WITH|UNDER)\s+[^\)]+\)//is; $view_infos{$view}{text} =~ s/\s*OF\s+XMLTYPE\s+[^\)]+\)//is; $view_infos{$view}{text} = $self->_format_view($view, $view_infos{$view}{text}); my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $view_infos{$view}{text}, 'VIEW'); $report_info{'Objects'}{$typ}{'cost_value'} += $cost; # Do not show view that just have to be tested next if (!$cost); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; # Show detail about views that might need manual rewritting $report_info{'Objects'}{$typ}{'detail'} .= "\L$view: $cost\E\n"; $report_info{full_view_details}{"\L$view\E"}{count} = $cost; foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { next if (!$cost_detail{$d}); $report_info{full_view_details}{"\L$view\E"}{info} .= "\t$d => $cost_detail{$d}"; $report_info{full_view_details}{"\L$view\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); $report_info{full_view_details}{"\L$view\E"}{info} .= "\n"; push(@{$report_info{full_view_details}{"\L$view\E"}{keywords}}, $d); } } } $report_info{'Objects'}{$typ}{'comment'} = "Views are fully supported but can use specific functions."; } elsif ($typ eq 'DATABASE LINK') { my $def_fdw = 'oracle_fdw'; $def_fdw = 'mysql_fdw' if ($self->{is_mysql}); $report_info{'Objects'}{$typ}{'comment'} = "Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions using $def_fdw."; if ($self->{estimate_cost}) { $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{'DATABASE LINK'}*$objects{$typ}); } } elsif ($typ eq 'JOB') { $report_info{'Objects'}{$typ}{'comment'} = "Job are not exported. You may set external cron job with them."; if ($self->{estimate_cost}) { $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{'JOB'}*$objects{$typ}); } } # Apply maximum cost per object type if (exists $Ora2Pg::PLSQL::MAX_SCORE{$typ} && $report_info{'Objects'}{$typ}{'cost_value'} > $Ora2Pg::PLSQL::MAX_SCORE{$typ}) { $report_info{'Objects'}{$typ}{'cost_value'} = $Ora2Pg::PLSQL::MAX_SCORE{$typ}; } $report_info{'total_cost_value'} += $report_info{'Objects'}{$typ}{'cost_value'}; $report_info{'Objects'}{$typ}{'cost_value'} = sprintf("%2.2f", $report_info{'Objects'}{$typ}{'cost_value'}); } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($idx, $num_total_obj, 25, '=', 'objects types', 'end of objects auditing.'), "\n"; } # DBA_AUDIT_TRAIL queries will not be count if no audit user is give if ($self->{audit_user}) { my $tbname = 'DBA_AUDIT_TRAIL'; $tbname = 'general_log' if ($self->{is_mysql}); $report_info{'Objects'}{'QUERY'}{'number'} = 0; $report_info{'Objects'}{'QUERY'}{'invalid'} = 0; $report_info{'Objects'}{'QUERY'}{'comment'} = "Normalized queries found in $tbname for user(s): $self->{audit_user}"; my %queries = $self->_get_audit_queries(); foreach my $q (sort {$a <=> $b} keys %queries) { $report_info{'Objects'}{'QUERY'}{'number'}++; my $sql_q = Ora2Pg::PLSQL::convert_plsql_code($self, $queries{$q}); if ($self->{estimate_cost}) { my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_q, 'QUERY'); $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'QUERY'}; $report_info{'Objects'}{'QUERY'}{'cost_value'} += $cost; $report_info{'total_cost_value'} += $cost; } } $report_info{'Objects'}{'QUERY'}{'cost_value'} = sprintf("%2.2f", $report_info{'Objects'}{'QUERY'}{'cost_value'}); } $report_info{'total_cost_value'} = sprintf("%2.2f", $report_info{'total_cost_value'}); # Display report in the requested format $self->_show_report(%report_info); } elsif ($type eq 'SHOW_SCHEMA') { # Get all tables information specified by the DBI method table_info $self->logit("Showing all schema...\n", 1); my $sth = $self->_schema_list() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { my $warning = ''; my $ret = $self->is_reserved_words($row[0]); if ($ret == 1) { $warning = " (Warning: '$row[0]' is a reserved word in PostgreSQL)"; } elsif ($ret == 2) { $warning = " (Warning: '$row[0]' object name with numbers only must be double quoted in PostgreSQL)"; } if (!$self->{is_mysql}) { $self->logit("SCHEMA $row[0]$warning\n", 0); } else { $self->logit("DATABASE $row[0]$warning\n", 0); } } $sth->finish(); } elsif ( ($type eq 'SHOW_TABLE') || ($type eq 'SHOW_COLUMN') ) { # Get all tables information specified by the DBI method table_info $self->logit("Showing table information...\n", 1); # Retrieve tables informations my %tables_infos = $self->_table_info(($type eq 'SHOW_TABLE') ? $self->{count_rows}: 0); # Retrieve column identity information $self->logit("Retrieving column identity information...\n", 1); %{ $self->{identity_info} } = $self->_get_identities(); # Retrieve all columns information my %columns_infos = (); if ($type eq 'SHOW_COLUMN') { %columns_infos = $self->_column_info('',$self->{schema}, 'TABLE'); foreach my $tb (keys %columns_infos) { foreach my $c (keys %{$columns_infos{$tb}}) { push(@{$self->{tables}{$tb}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); } } %columns_infos = (); # Look for encrypted columns %{$self->{encrypted_column}} = $self->_encrypted_columns('',$self->{schema}); # Retrieve index informations my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}); foreach my $tb (keys %{$indexes}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; } foreach my $tb (keys %{$idx_type}) { next if (!exists $tables_infos{$tb}); %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; } foreach my $idx (keys %{ $self->{tables}{$tb}{idx_type} }) { if ($self->{tables}{$tb}{idx_type}{$idx}{type} =~ /COLUMNSTORE/) { $self->{tables}{$tb}{columnstore} = 1; } } } # Get partition list to mark tables with partition. $self->logit("Looking to subpartition information...\n", 1); my %subpartitions_list = $self->_get_subpartitioned_table(); $self->logit("Looking to partitioned tables information...\n", 1); my %partitions = $self->_get_partitioned_table(%subpartitions_list); # Look for external tables my %externals = (); if (!$self->{is_mysql} && ($self->{db_version} !~ /Release 8/)) { $self->logit("Looking to external tables information...\n", 1); %externals = $self->_get_external_tables(); } # Ordering tables by name by default my @ordered_tables = sort { $a cmp $b } keys %tables_infos; if (lc($self->{data_export_order}) eq 'size') { @ordered_tables = sort { ($tables_infos{$b}{num_rows} || $tables_infos{$a}{num_rows}) ? $tables_infos{$b}{num_rows} <=> $tables_infos{$a}{num_rows} : $a cmp $b } keys %tables_infos; } # User provide the ordered list of table from a file elsif (-e $self->{data_export_order}) { if (open(my $tfh, '<', $self->{data_export_order})) { @ordered_tables = (); while (my $l = <$tfh>) { chomp($l); next if (!exists $self->{tables}{$!}); push(@ordered_tables, $l); } close($tfh); } else { $self->logit("FATAL: can't read file $self->{data_export_order} for ordering table export. $!\n", 0, 1); } } my @done = (); my $id = 0; # Set the table information for each class found my $i = 1; my $total_row_num = 0; foreach my $t (@ordered_tables) { # Jump to desired extraction if (grep(/^\Q$t\E$/, @done)) { $self->logit("Duplicate entry found: $t\n", 1); next; } else { push(@done, $t); } my $warning = ''; # Add a warning when the table name is > 63 character if (length($t) > 63) { $warning .= " (>63)"; } # Signal that the table use columnstore if ($self->{is_mssql} && $self->{tables}{$tb}{columnstore} == 1) { $warning .= " - storage: columnar"; } # Show compression type if ($self->{is_mssql} && exists $tables_infos{$t}{compressed} && $tables_infos{$t}{compressed} ne 'NONE') { $warning .= " - compression: $tables_infos{$t}{compressed}"; } # Set the number of partition if any if (exists $partitions{"\L$t\E"}) { my $upto = ''; $upto = 'up to ' if ($partitions{"\L$t\E"}{count} == 1048575); $warning .= " - $upto" . $partitions{"\L$t\E"}{count} . " " . $partitions{"\L$t\E"}{type} . " partitions"; $warning .= " with subpartitions" if ($partitions{"\L$t\E"}{composite}); } # Search for reserved keywords my $ret = $self->is_reserved_words($t); if ($ret == 1) { $warning .= " (Warning: '$t' is a reserved word in PostgreSQL)"; } elsif ($ret == 2) { $warning .= " (Warning: '$t' object name with numbers only must be double quoted in PostgreSQL)"; } $total_row_num += $tables_infos{$t}{num_rows}; # Show table information my $kind = ''; $kind = ' FOREIGN' if ($tables_infos{$t}{connection}); if ($tables_infos{$t}{partitioned}) { $kind = ' PARTITIONED'; } if (exists $externals{$t}) { $kind = ' EXTERNAL'; } if ($tables_infos{$t}{nologging}) { $kind .= ' UNLOGGED'; } if ($tables_infos{$t}{index_type}) { $warning .= " - Indexed type: $tables_infos{$t}{index_type}"; } my $tname = $t; if (!$self->{is_mysql}) { $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); $self->logit("[$i]$kind TABLE $tname (owner: $tables_infos{$t}{owner}, $tables_infos{$t}{num_rows} rows)$warning\n", 0); } else { $self->logit("[$i]$kind TABLE $tname ($tables_infos{$t}{num_rows} rows)$warning\n", 0); } # Set the fields information if ($type eq 'SHOW_COLUMN') { # Collect column's details for the current table with attempt to preserve column declaration order foreach my $k (sort { if (!$self->{reordering_columns}) { $self->{tables}{$t}{column_info}{$a}[11] <=> $self->{tables}{$t}{column_info}{$b}[11]; } else { my $tmpa = $self->{tables}{$t}{column_info}{$a}; $tmpa->[2] =~ s/\D//g; my $typa = $self->_sql_type($tmpa->[1], $tmpa->[2], $tmpa->[5], $tmpa->[6], $tmpa->[4]); $typa =~ s/\(.*//; my $tmpb = $self->{tables}{$t}{column_info}{$b}; $tmpb->[2] =~ s/\D//g; my $typb = $self->_sql_type($tmpb->[1], $tmpb->[2], $tmpb->[5], $tmpb->[6], $tmpb->[4]); $typb =~ s/\(.*//; $TYPALIGN{$typb} <=> $TYPALIGN{$typa}; } } keys %{$self->{tables}{$t}{column_info}}) { $warning = ''; # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE my $d = $self->{tables}{$t}{column_info}{$k}; $d->[2] =~ s/[^0-9\-\.]//g; my $type1 = $self->_sql_type($d->[1], $d->[2], $d->[5], $d->[6], $d->[4]); $type1 = "$d->[1], $d->[2]" if (!$type1); $warning .= " (numeric?)" if ($type1 =~ s/#$//); # Check if we need auto increment if ($d->[12] eq 'auto_increment' || $d->[12] eq '1') { if ($type1 !~ s/bigint/bigserial/) { if ($type1 !~ s/smallint/smallserial/) { $type1 =~ s/integer/serial/; $type1 =~ s/numeric.*/bigserial/; } } if ($type1 =~ /serial/) { $warning = " - Seq last value: $tables_infos{$t}{auto_increment}"; } } $type1 = $self->{'modify_type'}{"\L$t\E"}{"\L$k\E"} if (exists $self->{'modify_type'}{"\L$t\E"}{"\L$k\E"}); my $align = ''; my $len = $d->[2]; if (($d->[1] =~ /char/i) && ($d->[7] > $d->[2])) { $d->[2] = $d->[7]; } if (length($d->[0]) > 63) { $warning .= " (>63)"; } if ($d->[1] eq 'DATE') { $warning .= " (date?)"; } if ($self->{is_mssql} && $d->[16]) { $warning .= " [Masked with: $d->[17]]"; } $self->logit("\t$d->[0] : $d->[1]"); if ($d->[1] !~ /SDO_GEOMETRY/) { if ($d->[2] && !$d->[5] && $d->[1] !~ /\(\d+\)/) { $self->logit("($d->[2])"); } elsif ($d->[5] && ($d->[1] =~ /NUMBER/i) ) { $self->logit("($d->[5]"); $self->logit(",$d->[6]") if ($d->[6]); $self->logit(")"); } if ($self->{reordering_columns}) { my $typ = $type1; $typ =~ s/\(.*//; $align = " - typalign: $TYPALIGN{$typ}"; } } else { # 12:SRID,13:SDO_DIM,14:SDO_GTYPE # Set the dimension, array is (srid, dims, gtype) my $suffix = ''; if ($d->[13] == 3) { $suffix = 'Z'; } elsif ($d->[13] == 4) { $suffix = 'ZM'; } my $gtypes = ''; if (!$d->[14] || ($d->[14] =~ /,/) ) { $gtypes = $Ora2Pg::Oracle::ORA2PG_SDO_GTYPE{0}; } else { $gtypes = $d->[14]; } $type1 = "geometry($gtypes$suffix"; if ($d->[12]) { $type1 .= ",$d->[12]"; } $type1 .= ")"; $type1 .= " - $d->[14]" if ($d->[14] =~ /,/); } my $ret = $self->is_reserved_words($d->[0]); if ($ret == 1) { $warning .= " (Warning: '$d->[0]' is a reserved word in PostgreSQL)"; } elsif ($ret == 2) { $warning .= " (Warning: '$d->[0]' object name with numbers only must be double quoted in PostgreSQL)"; } elsif ($ret == 3) { $warning = " (Warning: '$d->[0]' is a system column in PostgreSQL)"; } # Check if this column should be replaced by a boolean following table/column name my $typlen = $d->[5]; $typlen ||= $d->[2]; if (grep(/^$d->[0]$/i, @{$self->{'replace_as_boolean'}{uc($t)}})) { $type1 = 'boolean'; } # Check if this column should be replaced by a boolean following type/precision elsif (exists $self->{'replace_as_boolean'}{uc($d->[1])} && ($self->{'replace_as_boolean'}{uc($d->[1])}[0] == $typlen)) { $type1 = 'boolean'; } # Autoincremented columns if (!$self->{schema} && $self->{export_schema}) { $d->[8] = "$d->[9].$d->[8]"; } if (exists $self->{identity_info}{$d->[8]}{$d->[0]}) { if ($self->{pg_supports_identity}) { $type1 = 'bigint' if ($self->{force_identity_bigint}); # Force bigint $type1 .= " GENERATED $self->{identity_info}{$d->[8]}{$d->[0]}{generation} AS IDENTITY"; if (exists $self->{identity_info}{$d->[8]}{$d->[0]}{options} && $self->{identity_info}{$d->[8]}{$d->[0]}{options} ne '') { # Adapt automatically the max value following the data type if ($type1 =~ /^(integer|int4|int)$/) { $self->{identity_info}{$d->[8]}{$d->[0]}{options} =~ s/ 9223372036854775807/ 2147483647/s; } $type1 .= " (" . $self->{identity_info}{$d->[8]}{$d->[0]}{options} . ')'; } } else { $type1 =~ s/bigint$/bigserial/; $type1 =~ s/smallint/smallserial/; $type1 =~ s/(integer|int)$/serial/; } } my $encrypted = ''; $encrypted = " [encrypted]" if (exists $self->{encrypted_column}{"$t.$k"}); my $virtual = ''; $virtual = " [virtual column]" if ($d->[10] eq 'YES' && $d->[4]); $self->logit(" => $type1$warning$align$virtual$encrypted\n"); } } $i++; } $self->logit("----------------------------------------------------------\n", 0); $self->logit("Total number of rows: $total_row_num\n\n", 0); # Looking for Global temporary tables my %global_tables = $self->_global_temp_table_info(); $self->logit("Global Temporary Tables:\n", 0); foreach my $k (sort keys %global_tables) { $self->logit("\t$k\n", 0); } $self->logit("\n\n", 0); $self->logit("Top $self->{top_max} of tables sorted by number of rows:\n", 0); $i = 1; foreach my $t (sort {$tables_infos{$b}{num_rows} <=> $tables_infos{$a}{num_rows}} keys %tables_infos) { next if ($tables_infos{$t}{num_rows} == 0); my $tname = $t; if (!$self->{is_mysql}) { $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); } $self->logit("\t[$i] TABLE $tname has $tables_infos{$t}{num_rows} rows\n", 0); $i++; last if ($i > $self->{top_max}); } $self->logit("Top $self->{top_max} of largest tables:\n", 0); $i = 1; if ($self->{is_mssql} || $self->{is_mysql}) { foreach my $t (sort {$tables_infos{$b}{size} <=> $tables_infos{$a}{size}} keys %tables_infos) { next if ($tables_infos{$t}{size} == 0); my $tname = $t; if (!$self->{is_mysql} && !$self->{is_mssql}) { $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); } $self->logit("\t[$i] TABLE $tname: $tables_infos{$t}{size} MB (" . $tables_infos{$t}{num_rows} . " rows)\n", 0); $i++; last if ($i > $self->{top_max}); } } else { # Because we avoid using JOIN when querying the Oracle catalog, look for all table size my %largest_table = $self->_get_largest_tables(); foreach my $t (sort { $largest_table{$b} <=> $largest_table{$a} } keys %largest_table) { last if ($i > $self->{top_max}); my $tname = $t; $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); $self->logit("\t[$i] TABLE $tname: $largest_table{$t} MB (" . $tables_infos{$t}{num_rows} . " rows)\n", 0); $i++; } } } } sub show_test_errors { my ($self, $lbl_type, @errors) = @_; print "[ERRORS \U$lbl_type\E COUNT]\n"; if ($#errors >= 0) { foreach my $msg (@errors) { print "DIFF: $msg\n"; } } else { if ($self->{pg_dsn}) { print "OK, Oracle and PostgreSQL have the same number of $lbl_type.\n"; } else { print "No PostgreSQL connection, can not check number of $lbl_type.\n"; } } } sub set_pg_relation_name { my ($self, $table) = @_; my $tbmod = $self->get_replaced_tbname($table); my $cmptb = $tbmod; $cmptb =~ s/"//g; my $orig = ''; $orig = " (origin: $table)" if (lc($cmptb) ne lc($table)); my $tbname = $tbmod; $tbname =~ s/[^"\.]+\.//; my $schm = $self->{schema}; $schm = $self->{pg_schema} if ($self->{pg_schema}); $schm =~ s/"//g; $tbname =~ s/"//g; $schm = $self->quote_object_name($schm); $tbname = $self->quote_object_name($tbname); if ($self->{pg_schema}) { if ($self->{preserve_case}) { return ($tbmod, $orig, $self->{pg_schema}, "\"$schm\".\"$tbname\""); } else { return ($tbmod, $orig, $self->{pg_schema}, "$schm.$tbname"); } } elsif ($self->{schema} && $self->{export_schema}) { if ($self->{preserve_case}) { return ($tbmod, $orig, $self->{schema}, "\"$schm\".\"$tbname\""); } else { return ($tbmod, $orig, $self->{schema}, "$schm.$tbname"); } } return ($tbmod, $orig, '', $tbname); } sub get_schema_condition { my ($self, $attrname, $local_schema) = @_; $attrname ||= 'n.nspname'; if ($local_schema && $self->{export_schema}) { return " AND lower($attrname) = quote_ident('\L$local_schema\E')"; } elsif ($self->{pg_schema} && $self->{export_schema}) { my $sql = " AND lower($attrname) IN ("; foreach my $s (split(/\s*,\s*/, $self->{pg_schema})) { $sql .= "quote_ident('\L$s\E'),"; } $sql =~ s/,$//; return $sql . ")"; } elsif ($self->{schema} && $self->{export_schema}) { return "AND lower($attrname) = quote_ident('\L$self->{schema}\E')"; } elsif ($self->{pg_schema}) { return "AND lower($attrname) = quote_ident('\L$self->{pg_schema}\E')"; } my $cond = " AND $attrname <> 'pg_catalog' AND $attrname <> 'information_schema' AND $attrname !~ '^pg_toast'"; return $cond; } sub _count_pg_rows { my ($self, $dbhdest, $lbl, $t, $num_rows) = @_; chomp($num_rows); my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); if ($self->{pg_dsn}) { my $err = ''; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); $self->logit("DEBUG: pid $$ looking for real row count for destination table $t...\n", 1); my $sql = "SELECT count(*) FROM $both;"; # if ($self->{preserve_case}) { # $sql = "SELECT count(*) FROM \"$schema\".\"$t\";"; # } my $s = $dbhdest->prepare($sql); if (not defined $s) { $err = "Table $both$orig does not exists in PostgreSQL database." if ($s->state eq '42P01'); } else { if (not $s->execute) { $err = "Table $both$orig does not exists in PostgreSQL database." if ($s->state eq '42P01'); } else { my $fh = new IO::File; $fh->open(">>${dirprefix}ora2pg_stdout_locker") or $self->logit("FATAL: can't write to ${dirprefix}ora2pg_stdout_locker, $!\n", 0, 1); flock($fh, 2) || die "FATAL: can't lock file ${dirprefix}ora2pg_stdout_locker\n"; print "$lbl:$t:$num_rows\n"; while ( my @row = $s->fetchrow()) { print "POSTGRES:$both$orig:$row[0]\n"; if ($row[0] != $num_rows) { $fh->print("Table $both$orig doesn't have the same number of line in source database ($num_rows) and in PostgreSQL ($row[0]).\n"); } last; } $s->finish(); $fh->close; } } if ($err) { my $fh = new IO::File; $fh->open(">>${dirprefix}ora2pg_stdout_locker") or $self->logit("FATAL: can't write to ${dirprefix}ora2pg_stdout_locker, $!\n", 0, 1); flock($fh, 2) || die "FATAL: can't lock file ${dirprefix}ora2pg_stdout_locker\n"; print "$lbl:$t:$num_rows\n"; $fh->print("$err\n"); $fh->close; } } } sub _table_row_count { my $self = shift; my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); # Get all tables information specified by the DBI method table_info $self->logit("Looking for real row count in source database and PostgreSQL tables...\n", 1); # Retrieve tables informations my %tables_infos = $self->_table_info($self->{count_rows}); #### # Test number of row in tables #### my @errors = (); print "\n"; print "[TEST ROWS COUNT]\n"; foreach my $t (sort keys %tables_infos) { if ($self->{parallel_tables} > 1) { spawn sub { my $dbhpg = $self->{dbhdest}->clone(); $self->_count_pg_rows($dbhpg, $lbl, $t, $tables_infos{$t}{num_rows}); $dbhpg->disconnect(); }; $parallel_tables_count++; # Wait for connection terminaison while ($parallel_tables_count > $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $parallel_tables_count--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } else { $self->_count_pg_rows($self->{dbhdest}, $lbl, $t, $tables_infos{$t}{num_rows}); } } # Wait for all child die if ($self->{parallel_tables} > 1) { while (scalar keys %RUNNING_PIDS > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { delete $RUNNING_PIDS{$kid}; } usleep(50000); } } my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); if (-e "${dirprefix}ora2pg_stdout_locker") { my $fh = new IO::File; $fh->open("${dirprefix}ora2pg_stdout_locker") or $self->logit("FATAL: can't read file ${dirprefix}ora2pg_stdout_locker, $!\n", 0, 1); @errors = <$fh>; $fh->close; unlink("${dirprefix}ora2pg_stdout_locker"); chomp @errors; } $self->show_test_errors('rows', @errors); } sub is_in_struct { my ($self, $t, $cn) = @_; if (!$self->{preserve_case}) { if (exists $self->{modify}{"\L$t\E"}) { return 0 if (!grep(/^\Q$cn\E$/i, @{$self->{modify}{"\L$t\E"}})); } elsif (exists $self->{exclude_columns}{"\L$t\E"}) { return 0 if (grep(/^\Q$cn\E$/i, @{$self->{exclude_columns}{"\L$t\E"}})); } } else { if (exists $self->{modify}{"$t"}) { return 0 if (!grep(/^\Q$cn\E$/i, @{$self->{modify}{"$t"}})); } elsif (exists $self->{exclude_columns}{"$t"}) { return 0 if (grep(/^\Q$cn\E$/i, @{$self->{exclude_columns}{"$t"}})); } } return 1; } sub _col_count { my ($self, $table, $schema) = @_; my %col_count = (); if ($self->{is_mysql}) { %col_count = Ora2Pg::MySQL::_col_count($self, $table, $schema); } elsif ($self->{is_mssql}) { %col_count = Ora2Pg::MSSQL::_col_count($self, $table, $schema); } else { %col_count = Ora2Pg::Oracle::_col_count($self, $table, $schema); } return %col_count; } sub _test_table { my $self = shift; my @errors = (); # Get all tables information specified by the DBI method table_info $self->logit("Looking for objects count related to source database and PostgreSQL tables...\n", 1); # Retrieve tables informations my %tables_infos = $self->_table_info(); my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); #### # Test number of column in tables #### print "[TEST COLUMNS COUNT]\n"; my %col_count = $self->_col_count('', $self->{schema}); $schema_cond = $self->get_schema_condition('pg_class.relnamespace::regnamespace::text'); my $sql = qq{ SELECT upper(pg_namespace.nspname||'.'||pg_class.relname), pg_attribute.attname FROM pg_attribute JOIN pg_class ON (pg_class.oid=pg_attribute.attrelid) JOIN pg_namespace ON (pg_class.relnamespace=pg_namespace.oid) WHERE pg_class.relkind IN ('r', 'p') AND pg_attribute.attnum > 0 AND NOT pg_attribute.attisdropped $schema_cond ORDER BY pg_attribute.attnum }; my %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about indexes."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); push(@{$pgret{$row[0]}}, $row[1]); } $s->finish; } my %pgcount = (); foreach my $t (keys %pgret) { $pgcount{$t} = $#{$pgret{$t}} + 1; } my @tables_names = keys %tables_infos; foreach my $t (sort keys %col_count) { next if (!grep(/^\Q$t\E$/i, @tables_names)); print "$lbl:$t:$col_count{$t}\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); my $tbname = uc($both); $tbname =~ s/"//g; $pgcount{$tbname} ||= 0; print "POSTGRES:$both$orig:", $pgcount{$tbname}, "\n"; if ($pgcount{$tbname} != $col_count{$t}) { push(@errors, "Table $both$orig doesn't have the same number of columns in source database ($col_count{$t}) and in PostgreSQL (" . $pgcount{$tbname} . ")."); push(@errors, "\tPostgreSQL modified struct: $both$orig(" . join(',', @{$pgret{$tbname}}) . ")"); } } } %pgcount = (); $self->show_test_errors('columns', @errors); @errors = (); #### # Test number of index in tables #### print "[TEST INDEXES COUNT]\n"; my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = undef; # special case for MySQL, FIXME: we should use $self->_get_indexes() for both if ($self->{is_mysql}) { $indexes = Ora2Pg::MySQL::_count_indexes($self, '', $self->{schema}); } else { ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('', $self->{schema}, 1); } $schema_cond = $self->get_schema_condition('tn.nspname'); my $exclude_unique = ''; $exclude_unique = 'AND NOT i.indisunique' if ($self->{is_mssql}); $sql = qq{SELECT tn.nspname||'.'||t.relname, count(*) FROM pg_index i JOIN pg_class c on c.oid = i.indexrelid JOIN pg_namespace n on n.oid = c.relnamespace JOIN pg_class t on t.oid = i.indrelid JOIN pg_namespace tn on tn.oid = t.relnamespace WHERE 1=1 $exclude_unique $schema_cond GROUP BY tn.nspname, t.relname }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about indexes."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is no indexes in a table foreach my $t (keys %tables_infos) { $indexes->{$t} = {} if (not exists $indexes->{$t}); } foreach my $t (sort keys %{$indexes}) { next if (!exists $tables_infos{$t}); my $numixd = scalar keys %{$indexes->{$t}}; print "$lbl:$t:$numixd\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $numixd) { push(@errors, "Table $both$orig doesn't have the same number of indexes in source database ($numixd) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } $self->show_test_errors('indexes', @errors); @errors = (); #### # Test unique constraints (excluding primary keys) #### print "\n"; print "[TEST UNIQUE CONSTRAINTS COUNT]\n"; my %unique_keys = $self->_unique_key('',$self->{schema},'U'); $schema_cond = $self->get_schema_condition('tn.nspname'); $exclude_unique = ''; $exclude_unique = 'AND i.indisunique' if ($self->{is_mssql}); $sql = qq{SELECT tn.nspname||'.'||t.relname, count(*) FROM pg_constraint c JOIN pg_class t on t.oid = c.conrelid JOIN pg_namespace tn on tn.oid = c.connamespace WHERE 1=1 $exclude_unique $schema_cond AND c.contype = 'u' AND c.conindid > 1 GROUP BY tn.nspname, t.relname }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about unique constraints."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is not unique key in a table foreach my $t (keys %tables_infos) { $unique_keys{$t} = {} if (not exists $unique_keys{$t}); } foreach my $t (sort keys %unique_keys) { next if (!exists $tables_infos{$t}); my $numixd = scalar keys %{$unique_keys{$t}}; if ($self->{is_mysql}) { foreach my $k (keys %{$unique_keys{$t}}) { $numixd-- if ($unique_keys{$t}{$k}{type} eq 'P'); } } print "$lbl:$t:$numixd\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $numixd) { push(@errors, "Table $both$orig doesn't have the same number of unique constraints in source database ($numixd) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } $self->show_test_errors('unique constraints', @errors); @errors = (); #### # Test primary keys only #### print "\n"; print "[TEST PRIMARY KEYS COUNT]\n"; %unique_keys = $self->_unique_key('',$self->{schema},'P'); $schema_cond = $self->get_schema_condition('pg_class.relnamespace::regnamespace::text'); $sql = qq{ SELECT schemaname||'.'||tablename, count(*) FROM pg_indexes JOIN pg_class ON (pg_class.relname=pg_indexes.indexname AND pg_class.relnamespace=pg_indexes.schemaname::regnamespace::oid) JOIN pg_constraint ON (pg_constraint.conname=pg_class.relname AND pg_constraint.connamespace=pg_class.relnamespace) WHERE pg_constraint.contype = 'p' $schema_cond GROUP BY schemaname,tablename }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about primary keys."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is not unique key in a table foreach my $t (keys %tables_infos) { $unique_keys{$t} = {} if (not exists $unique_keys{$t}); } foreach my $t (sort keys %unique_keys) { next if (!exists $tables_infos{$t}); my $nbpk = 0; foreach my $c (keys %{$unique_keys{$t}}) { $nbpk++ if ($unique_keys{$t}{$c}{type} eq 'P'); } print "$lbl:$t:$nbpk\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbpk) { push(@errors, "Table $both$orig doesn't have the same number of primary keys in source database ($nbpk) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } %unique_keys = (); $self->show_test_errors('primary keys', @errors); @errors = (); #### # Test check constraints #### my %nbnotnull = {}; # will be used in the NOT NULL constraint count as based on a CHECK constraints print "\n"; print "[TEST CHECK CONSTRAINTS COUNT]\n"; my %check_constraints = $self->_check_constraint('',$self->{schema}); $schema_cond = $self->get_schema_condition('n.nspname'); $sql = qq{ SELECT CASE WHEN regexp_count(r.conrelid::regclass::text, n.nspname||'.') > 0 THEN r.conrelid::regclass::text ELSE n.nspname::regnamespace||'.'||r.conrelid::regclass END, count(*) FROM pg_catalog.pg_constraint r JOIN pg_class c ON (r.conrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) WHERE r.contype = 'c' $schema_cond GROUP BY n.nspname,r.conrelid }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about check constraints."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is not unique key in a table foreach my $t (keys %tables_infos) { $check_constraints{$t}{constraint} = {} if (not exists $check_constraints{$t}); } foreach my $t (sort keys %check_constraints) { next if (!exists $tables_infos{$t}); my $nbcheck = 0; foreach my $cn (keys %{$check_constraints{$t}{constraint}}) { if ($check_constraints{$t}{constraint}{$cn}{condition} =~ /^[^\s]+\s+IS\s+NOT\s+NULL$/i) { $nbnotnull{$t}{$check_constraints{$t}{constraint}{$cn}{condition}} = 1; } else { $nbcheck++; } } print "$lbl:$t:$nbcheck\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbcheck) { push(@errors, "Table $both$orig doesn't have the same number of check constraints in source database ($nbcheck) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } %check_constraints = (); $self->show_test_errors('check constraints', @errors); @errors = (); #### # Test NOT NULL constraints #### print "\n"; print "[TEST NOT NULL CONSTRAINTS COUNT]\n"; my %column_infos = $self->_column_attributes('', $self->{schema}, 'TABLE'); $schema_cond = $self->get_schema_condition('n.nspname'); $sql = qq{ SELECT n.nspname||'.'||e.oid::regclass, count(*) FROM pg_catalog.pg_attribute a JOIN pg_class e ON (e.oid=a.attrelid) JOIN pg_namespace n ON (e.relnamespace=n.oid) WHERE a.attnum > 0 AND e.relkind IN ('r') AND NOT a.attisdropped AND a.attnotnull $schema_cond GROUP BY n.nspname,e.oid }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about not null constraints using query: $sql"); $self->show_test_errors('not null constraints', @errors); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.([^\.]+\.)/$1/; # remove possible duplicate schema prefix $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } foreach my $t (sort keys %column_infos) { next if (!exists $tables_infos{$t}); my $nbnull = 0; foreach my $cn (keys %{$column_infos{$t}}) { next if (!$self->is_in_struct($t, $cn)); if ($column_infos{$t}{$cn}{nullable} =~ /^N/) { $nbnull++; } } # Append the CHECK not null constraints if any if (exists $nbnotnull{$t}) { $nbnull = scalar keys %{ $nbnotnull{$t} } if (scalar keys %{ $nbnotnull{$t} } > $nbnull); } print "$lbl:$t:$nbnull\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbnull) { push(@errors, "Table $both$orig doesn't have the same number of not null constraints in source database ($nbnull) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } $self->show_test_errors('not null constraints', @errors); @errors = (); #### # Test column default values #### print "\n"; print "[TEST COLUMN DEFAULT VALUE COUNT]\n"; $schema_cond = $self->get_schema_condition('n.nspname'); # SELECT n.nspname||'.'||e.oid::regclass, $sql = qq{ SELECT n.nspname||'.'||e.oid::regclass, count((SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128) FROM pg_catalog.pg_attrdef d WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef)) "default value" FROM pg_catalog.pg_attribute a JOIN pg_class e ON (e.oid=a.attrelid) JOIN pg_namespace n ON (e.relnamespace=n.oid) WHERE a.attnum > 0 AND NOT a.attisdropped $schema_cond GROUP BY n.nspname,e.oid }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about column default values."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.([^\.]+\.)/$1/; # remove possible duplicate schema prefix $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } my @seqs = (); # MySQL do not have sequences but we cound autoincrement column as sequences if ($self->{is_mysql}) { @seqs = Ora2Pg::MySQL::_count_sequences($self); } foreach my $t (sort keys %column_infos) { next if (!exists $tables_infos{$t}); my $nbdefault = 0; foreach my $cn (keys %{$column_infos{$t}}) { next if (!$self->is_in_struct($t, $cn)); if ($column_infos{$t}{$cn}{default} ne '' && uc($column_infos{$t}{$cn}{default}) ne 'NULL' # identity column && ( $column_infos{$t}{$cn}{default} !~ /ISEQ\$\$_.*nextval/i || $self->{is_mysql} || !$self->{pg_supports_identity}) ) { $nbdefault++; } } if (grep(/^$t$/i, @seqs)) { $nbdefault++; } print "$lbl:$t:$nbdefault\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbdefault) { push(@errors, "Table $both$orig doesn't have the same number of column default value in source database ($nbdefault) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } %column_infos = (); $self->show_test_errors('column default value', @errors); @errors = (); #### # Test identity columns #### if ($self->{is_mysql} || !$self->{pg_supports_identity}) { print "\n"; print "[TEST IDENTITY COLUMN COUNT]\n"; $schema_cond = $self->get_schema_condition('n.nspname'); $sql = qq{ SELECT e.oid::regclass, count((SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128) FROM pg_catalog.pg_attrdef d WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef)) "default value" FROM pg_catalog.pg_attribute a JOIN pg_class e ON (e.oid=a.attrelid) JOIN pg_namespace n ON (e.relnamespace=n.oid) WHERE a.attnum > 0 AND NOT a.attisdropped AND a.attidentity IN ('a', 'd') $schema_cond GROUP BY e.oid }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about identity columns."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } @seqs = (); # MySQL do not have sequences but we cound autoincrement column as sequences if ($self->{is_mysql}) { @seqs = Ora2Pg::MySQL::_count_sequences($self); } foreach my $t (sort keys %column_infos) { next if (!exists $tables_infos{$t}); my $nbidty = 0; foreach my $cn (keys %{$column_infos{$t}}) { next if (!$self->is_in_struct($t, $cn)); if ($column_infos{$t}{$cn}{default} =~ /ISEQ\$\$_.*nextval/i) { $nbidty++; } } if (grep(/^$t$/i, @seqs)) { $nbidty++; } print "$lbl:$t:$nbidty\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbidty) { push(@errors, "Table $both$orig doesn't have the same number of identity column in source database ($nbidty) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } %column_infos = (); $self->show_test_errors('column default value', @errors); @errors = (); } %column_infos = (); #### # Test foreign keys #### print "\n"; print "[TEST FOREIGN KEYS COUNT]\n"; my ($foreign_link, $foreign_key) = $self->_foreign_key('',$self->{schema}); $schema_cond = $self->get_schema_condition('n.nspname'); $sql = qq{ SELECT CASE WHEN regexp_count(r.conrelid::regclass::text, n.nspname||'.') > 0 THEN r.conrelid::regclass::text ELSE n.nspname::regnamespace||'.'||r.conrelid::regclass END, count(*) FROM pg_catalog.pg_constraint r JOIN pg_class c ON (r.conrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) WHERE r.contype = 'f' $schema_cond GROUP BY n.nspname,r.conrelid }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about foreign keys constraints."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.([^\.]+\.)/$1/; # remove possible duplicate schema prefix $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is not unique key in a table foreach my $t (keys %tables_infos) { $foreign_link->{$t} = {} if (not exists $foreign_link->{$t}); } foreach my $t (sort keys %{$foreign_link}) { next if (!exists $tables_infos{$t}); my $nbfk = scalar keys %{$foreign_link->{$t}}; print "$lbl:$t:$nbfk\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbfk) { push(@errors, "Table $both$orig doesn't have the same number of foreign key constraints in source database ($nbfk) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } $self->show_test_errors('foreign keys', @errors); @errors = (); $foreign_link = undef; $foreign_key = undef; #### # Test partitions #### print "\n"; print "[TEST PARTITION COUNT]\n"; my %partitions = $self->_get_partitioned_table(); $schema_cond = $self->get_schema_condition('nmsp_parent.nspname'); $sql = qq{ SELECT nmsp_parent.nspname || '.' || parent.relname, COUNT(*) FROM pg_inherits JOIN pg_class parent ON pg_inherits.inhparent = parent.oid JOIN pg_class child ON pg_inherits.inhrelid = child.oid JOIN pg_namespace nmsp_parent ON nmsp_parent.oid = parent.relnamespace JOIN pg_namespace nmsp_child ON nmsp_child.oid = child.relnamespace WHERE child.relkind = 'r' $schema_cond GROUP BY nmsp_parent.nspname || '.' || parent.relname }; my %pg_part = (); if ($self->{pg_dsn}) { $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about PARTITION."); next; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pg_part{$row[0]} = $row[1]; } $s->finish(); } foreach my $t (sort keys %tables_infos) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); next if (!exists $partitions{$t} && !exists $pg_part{$tbmod}); $partitions{$t}{count} ||= 0; print "$lbl:$t:$partitions{$t}{count}\n"; $pg_part{$tbmod} ||= 0; print "POSTGRES:$both$orig:$pg_part{$tbmod}\n"; if ($pg_part{$tbmod} != $partitions{$t}{count}) { push(@errors, "Table $both$orig doesn't have the same number of partitions in source database ($partitions{$t}{count}) and in PostgreSQL ($pg_part{$tbmod})."); } } $self->show_test_errors('PARTITION', @errors); @errors = (); %partitions = (); print "\n"; print "[TEST TABLE COUNT]\n"; my $nbobj = scalar keys %tables_infos; $schema_cond = $self->get_schema_condition(); $sql = qq{ SELECT count(*) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r', 'p') AND NOT c.relispartition AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') $schema_cond }; print "$lbl:TABLE:$nbobj\n"; if ($self->{pg_dsn}) { $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about $obj_type."); next; } while ( my @row = $s->fetchrow()) { print "POSTGRES:TABLE:$row[0]\n"; if ($row[0] != $nbobj) { push(@errors, "TABLE does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); } last; } $s->finish(); } $self->show_test_errors('TABLE', @errors); @errors = (); #### # Test triggers #### print "\n"; print "[TEST TABLE TRIGGERS COUNT]\n"; my %triggers = $self->_list_triggers(); $schema_cond = $self->get_schema_condition(); $sql = qq{ SELECT n.nspname||'.'||c.relname, count(*) FROM pg_catalog.pg_trigger t JOIN pg_class c ON (t.tgrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) WHERE (NOT t.tgisinternal OR (t.tgisinternal AND t.tgenabled = 'D')) $schema_cond GROUP BY n.nspname,c.relname }; %pgret = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about table triggrers."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; } # Initialize when there is not unique key in a table foreach my $t (keys %tables_infos) { $triggers{$t} = () if (not exists $triggers{$t}); } foreach my $t (sort keys %triggers) { next if (!exists $tables_infos{$t}); my $nbtrg = @{$triggers{$t}}; print "$lbl:$t:$nbtrg\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); $pgret{"\U$both\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both\E"}, "\n"; if ($pgret{"\U$both\E"} != $nbtrg) { push(@errors, "Table $both$orig doesn't have the same number of triggers in source database ($nbtrg) and in PostgreSQL (" . $pgret{"\U$both\E"} . ")."); } } } $s->finish() if ($self->{pg_dsn}); $self->show_test_errors('table triggers', @errors); @errors = (); print "\n"; print "[TEST TRIGGER COUNT]\n"; $nbobj = 0; foreach my $t (keys %triggers) { next if (!exists $tables_infos{$t}); $nbobj += $#{$triggers{$t}}+1; } $schema_cond = $self->get_schema_condition(); $sql = qq{ SELECT count(*) FROM pg_catalog.pg_trigger t JOIN pg_class c ON (c.oid = t.tgrelid) JOIN pg_namespace n ON (c.relnamespace=n.oid) WHERE (NOT t.tgisinternal OR (t.tgisinternal AND t.tgenabled = 'D')) $schema_cond }; print "$lbl:TRIGGER:$nbobj\n"; if ($self->{pg_dsn}) { $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about $obj_type."); next; } while ( my @row = $s->fetchrow()) { print "POSTGRES:TRIGGER:$row[0]\n"; if ($row[0] != $nbobj) { push(@errors, "TRIGGER does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); } last; } $s->finish(); } $self->show_test_errors('TRIGGER', @errors); @errors = (); } sub _unitary_test_views { my $self = shift; # Get all tables information specified by the DBI method table_info $self->logit("Unitary test of views between source database and PostgreSQL...\n", 1); # First of all extract all views from PostgreSQL database my $schema_clause = $self->get_schema_condition(); my $sql = qq{ SELECT c.relname,n.nspname FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = 'v' AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend WHERE refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND objid = c.oid AND deptype = 'e') $schema_clause }; my %list_views = (); if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about views."); next; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $list_views{$row[0]} = $self->{schema} || $row[1]; } $s->finish(); } my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); print "[UNITARY TEST OF VIEWS]\n"; foreach my $v (sort keys %list_views) { # Execute init settings if any # Count rows returned by all view on the source database my $vname = $v; $sql = "SELECT count(*) FROM $list_views{$v}.$v"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 0); my @row = $sth->fetchrow(); my $ora_ct = $row[0]; print "$lbl:$vname:", join('|', @row), "\n"; $sth->finish; # Execute view in the PostgreSQL database $vname = "$list_views{$v}.$v" if ($self->{export_schema}); $vname = "$self->{pg_schema}.$v" if ($self->{pg_schema}); $sql = "SELECT count(*) FROM " .$self->quote_object_name($vname); $sth = $self->{dbhdest}->prepare($sql); if (not defined $sth) { $self->logit("ERROR: " . $self->{dbhdest}->errstr . "\n", 0, 0); next; } $sth->execute or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 0); @row = $sth->fetchrow(); $sth->finish; my $pg_ct = $row[0]; print "POSTGRES:$vname:", join('|', @row), "\n"; if ($pg_ct != $ora_ct) { print "ERROR: view $v returns different row count [oracle: $ora_ct, postgresql: $pg_ct]\n"; } } } sub _count_object { my $self = shift; my $obj_type = shift; # Get all tables information specified by the DBI method table_info $self->logit("Looking for source database and PostgreSQL objects count...\n", 1); my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); my $schema_clause = $self->get_schema_condition(); my $nbobj = 0; my $sql = ''; if ($obj_type eq 'VIEW') { my %obj_infos = $self->_get_views(); $nbobj = scalar keys %obj_infos; $sql = qq{ SELECT count(*) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('v','') AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') $schema_clause }; } elsif ($obj_type eq 'MVIEW') { my %obj_infos = $self->_get_materialized_views(); $nbobj = scalar keys %obj_infos; $sql = qq{ SELECT count(*) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('m','') AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') $schema_clause }; } elsif ($obj_type eq 'SEQUENCE') { my $obj_infos = {}; if (!$self->{is_mysql}) { $obj_infos = $self->_get_sequences(); } else { # MySQL do not have sequences but we cound autoincrement column as sequences $obj_infos = Ora2Pg::MySQL::_count_sequences($self); } $nbobj = scalar keys %$obj_infos; $sql = qq{ SELECT count(*) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('S','') AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') AND NOT EXISTS (SELECT 1 FROM pg_attribute a WHERE NOT a.attisdropped AND a.attidentity IN ('a', 'd') AND c.oid = pg_get_serial_sequence(a.attrelid::regclass::text, a.attname)::regclass::oid) $schema_clause }; } elsif ($obj_type eq 'TYPE') { my $obj_infos = $self->_get_types(); $nbobj = scalar @{$obj_infos}; $schema_clause .= " AND pg_catalog.pg_type_is_visible(t.oid)" if ($schema_clause =~ /information_schema/); $sql = qq{ SELECT count(*) FROM pg_catalog.pg_type t LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace WHERE (t.typrelid = 0 OR (SELECT c.relkind = 'c' FROM pg_catalog.pg_class c WHERE c.oid = t.typrelid)) AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_type el WHERE el.oid = t.typelem AND el.typarray = t.oid) AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = t.oid AND d.deptype = 'e') $schema_clause }; } elsif ($obj_type eq 'FDW') { my %obj_infos = $self->_get_external_tables(); $nbobj = scalar keys %obj_infos; $sql = qq{ SELECT count(*) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('f','') AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') $schema_clause }; } else { return; } print "\n"; print "[TEST $obj_type COUNT]\n"; if ($self->{is_mysql} && ($obj_type eq 'SEQUENCE')) { print "$lbl:AUTOINCR:$nbobj\n"; } else { print "$lbl:$obj_type:$nbobj\n"; } if ($self->{pg_dsn}) { my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about $obj_type."); next; } while ( my @row = $s->fetchrow()) { print "POSTGRES:$obj_type:$row[0]\n"; if ($row[0] != $nbobj) { push(@errors, "\U$obj_type\E does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); } last; } $s->finish(); } $self->show_test_errors($obj_type, @errors); @errors = (); } sub _test_function { my $self = shift; my @errors = (); $self->logit("Looking for functions count related to source database and PostgreSQL functions...\n", 1); my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); #### # Test number of function #### print "\n"; print "[TEST FUNCTION COUNT]\n"; my @fct_infos = $self->_list_all_functions(); my $schema_clause = $self->get_schema_condition(); if ($self->{package_as_schema}) { my %processed_pkgs; foreach my $f (@fct_infos) { my ($pkg, $fct) = split(/\./, $f); next if $processed_pkgs{$pkg}++; # Skip if package already processed $schema_clause .= ") OR (lower(n.nspname) = quote_ident('\L$pkg\E')"; } if (scalar keys %processed_pkgs > 0) { $schema_clause =~ s/^(\s*AND\s+)/$1\(/; $schema_clause =~ s/$/\)/; } } $sql = qq{ SELECT n.nspname,proname,prorettype FROM pg_catalog.pg_proc p LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace LEFT JOIN pg_catalog.pg_type t ON t.oid=p.prorettype WHERE t.typname <> 'trigger' AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = p.oid AND d.deptype = 'e') $schema_clause }; my $nbobj = $#fct_infos + 1; print "$lbl:FUNCTION:$nbobj\n"; if ($self->{pg_dsn}) { $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about $obj_type."); next; } my $pgfct = 0; my %pg_function = (); while ( my @row = $s->fetchrow()) { $pgfct++; my $fname = $row[1]; if ($row[0] ne 'public') { $fname = $row[0] . '.' . $row[1]; } $pg_function{lc($fname)} = 1; } print "POSTGRES:FUNCTION:$pgfct\n"; if ($pgfct != $nbobj) { push(@errors, "FUNCTION does not have the same count in source database ($nbobj) and in PostgreSQL ($pgfct)."); } $s->finish(); # search for missing funtions foreach my $f (@fct_infos) { my $found = 0; foreach my $pgf (keys %pg_function) { $found = 1, last if (lc($f) eq lc($pgf)); if ($f !~ /\./) { $found = 1, last if ($pgf =~ /^[^\.]+\.\Q$f\E$/i); } else { $found = 1, last if ($pgf =~ /^\Q$f\E$/i); } } push(@errors, "Function $f is missing in PostgreSQL database.") if (!$found); } # search for additional functions foreach my $pgf (keys %pg_function) { my $found = 0; foreach my $f (@fct_infos) { $found = 1, last if (lc($f) eq lc($pgf)); if ($f !~ /\./) { $found = 1, last if ($pgf =~ /^[^\.]+\.\Q$f\E$/i); } else { $found = 1, last if ($pgf =~ /^\Q$f\E$/i); } } push(@errors, "Function $pgf is in addition in PostgreSQL database.") if (!$found); } } $self->show_test_errors('FUNCTION', @errors); @errors = (); print "\n"; } sub _test_seq_values { my $self = shift; my @errors = (); $self->logit("Looking for last values related to source database and PostgreSQL sequences...\n", 1); my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); #### # Test number of function #### print "\n"; print "[TEST SEQUENCE VALUES]\n"; my $obj_infos = {}; if (!$self->{is_mysql}) { $obj_infos = $self->_get_sequences(); } else { # MySQL do not have sequences but we cound autoincrement column as sequences $obj_infos = Ora2Pg::MySQL::_count_sequences($self); } my %pgret = (); if ($self->{pg_dsn}) { # create a function to extract the last value of all sequences my $fqdn = ''; $fqdn = "$self->{pg_schema}\." if ($self->{pg_schema}); my $sql = qq{ CREATE OR REPLACE FUNCTION ${fqdn}get_sequence_last_values() RETURNS TABLE(seqname text,val bigint) AS \$\$ DECLARE seq_name varchar(128); BEGIN FOR seq_name in SELECT relnamespace::regnamespace::text || '.' || quote_ident(relname::text) FROM pg_class c WHERE (relkind = 'S') AND NOT EXISTS (SELECT 1 FROM pg_catalog.pg_depend d WHERE d.refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass AND d.objid = c.oid AND d.deptype = 'e') LOOP RETURN QUERY EXECUTE 'SELECT ' || quote_literal(seq_name) || ',last_value FROM ' || seq_name; END LOOP; RETURN; END \$\$ LANGUAGE 'plpgsql'; }; $self->{dbhdest}->do($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); my $s = $self->{dbhdest}->prepare("SELECT * FROM ${fqdn}get_sequence_last_values()") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about last values of sequences."); return; } while ( my @row = $s->fetchrow()) { $row[0] =~ s/^[^\.]+\.// if (!$self->{export_schema}); $pgret{"\U$row[0]\E"} = $row[1]; } $s->finish; $self->{dbhdest}->do("DROP FUNCTION ${fqdn}get_sequence_last_values") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } foreach my $r (sort keys %$obj_infos) { $r =~ s/^[^\.]+\.// if (!$self->{export_schema}); print "$lbl:$r:$obj_infos->{$r}->[4]\n"; if ($self->{pg_dsn}) { my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($r); $pgret{"\U$both$orig\E"} ||= 0; print "POSTGRES:$both$orig:", $pgret{"\U$both$orig\E"}, "\n"; if ($pgret{"\U$both$orig\E"} != $obj_infos->{$r}->[4]) { push(@errors, "Sequence $both$orig doesn't have the same value in source database ($obj_infos->{$r}->[4]) and in PostgreSQL (" . $pgret{"\U$both$orig\E"} . "). Verify +/- cache size: $obj_infos->{$r}->[5]."); } } } $self->show_test_errors('sequence values', @errors); @errors = (); print "\n"; } =head2 _get_version This function retrieves the Oracle version information =cut sub _get_version { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_version($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_version($self); } else { return Ora2Pg::Oracle::_get_version($self); } } =head2 _get_database_size This function retrieves the size of the Oracle database in MB =cut sub _get_database_size { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_database_size($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_database_size($self); } else { return Ora2Pg::Oracle::_get_database_size($self); } } =head2 _get_objects This function retrieves all object the Oracle information except SYNONYM and temporary objects =cut sub _get_objects { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_objects($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_objects($self); } else { return Ora2Pg::Oracle::_get_objects($self); } } sub _list_all_functions { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_list_all_functions($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_list_all_functions($self); } else { return Ora2Pg::Oracle::_list_all_functions($self); } } =head2 _schema_list This function retrieves all Oracle-native user schema. Returns a handle to a DB query statement. =cut sub _schema_list { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_schema_list($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_schema_list($self); } else { return Ora2Pg::Oracle::_schema_list($self); } } =head2 _table_exists This function return the table name if the given table exists else returns a empty string. =cut sub _table_exists { my ($self, $schema, $table) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_table_exists($self, $schema, $table); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_table_exists($self, $schema, $table); } else { return Ora2Pg::Oracle::_table_exists($self, $schema, $table); } } =head2 _get_largest_tables This function retrieves the list of largest table of the Oracle database in MB =cut sub _get_largest_tables { my $self = shift; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_largest_tables($self); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_largest_tables($self); } else { return Ora2Pg::Oracle::_get_largest_tables($self); } } =head2 _get_encoding This function retrieves the Oracle database encoding Returns a handle to a DB query statement. =cut sub _get_encoding { my ($self, $dbh) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_get_encoding($self, $self->{dbh}); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_get_encoding($self, $self->{dbh}); } else { return Ora2Pg::Oracle::_get_encoding($self, $self->{dbh}); } } =head2 _compile_schema This function force Oracle database to compile a schema and validate or invalidate PL/SQL code. When parameter $schema is the name of a schema, only this schema is recompiled When parameter $schema is equal to 1 and SCHEMA directive is set, only this schema is recompiled When parameter $schema is equal to 1 and SCHEMA directive is unset, all schema will be recompiled =cut sub _compile_schema { my ($self, $schema) = @_; my @to_compile = (); if ($schema and ($schema =~ /[a-z]/i)) { push(@to_compile, $schema); } elsif ($schema and $self->{schema}) { push(@to_compile, $self->{schema}); } elsif ($schema) { my $sth = $self->_schema_list() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@to_compile, $row[0]); } $sth->finish(); } if ($#to_compile >= 0 && $self->{type} !~ /^SHOW_/i) { foreach my $schm (@to_compile) { $self->logit("Force Oracle to compile schema $schm before code extraction\n", 1); my $sql = "BEGIN\nDBMS_UTILITY.compile_schema(schema => '$schm', compile_all => FALSE);\nEND;"; my $sth = $self->{dbh}->do($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . ", SQL: $sql\n", 0, 1); } } } =head2 _datetime_format This function force Oracle database to format the time correctly =cut sub _datetime_format { my ($self, $dbh) = @_; $dbh = $self->{dbh} if (!$dbh); if ($self->{enable_microsecond}) { my $dim = 6; $dim = '' if ($self->{db_version} =~ /Release [89]/); my $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS.FF$dim'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } else { my $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } my $sth = $dbh->do("ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); if ($self->{enable_microsecond}) { my $dim = 6; $dim = '' if ($self->{db_version} =~ /Release [89]/); $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT='YYYY-MM-DD HH24:MI:SS.FF$dim TZH:TZM'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } else { $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT='YYYY-MM-DD HH24:MI:SS TZH:TZM'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } } sub _numeric_format { my ($self, $dbh) = @_; $dbh = $self->{dbh} if (!$dbh); my $sth = $dbh->do("ALTER SESSION SET NLS_NUMERIC_CHARACTERS = '.,'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } sub _ora_initial_command { my ($self, $dbh) = @_; return if ($#{ $self->{ora_initial_command} } < 0); $dbh = $self->{dbh} if (!$dbh); # Lookup if the user have provided some sessions settings foreach my $q (@{$self->{ora_initial_command}}) { next if (!$q); $self->logit("DEBUG: executing initial command to Oracle: $q\n", 1); my $sth = $dbh->do($q) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } } sub _pg_initial_command { my ($self, $dbh) = @_; return if ($#{ $self->{pg_initial_command} } < 0); $dbh = $self->{dbhdest} if (!$dbh); # Lookup if the user have provided some sessions settings foreach my $q (@{$self->{pg_initial_command}}) { $self->logit("DEBUG: executing initial command to PostgreSQL: $q\n", 1); my $sth = $dbh->do($q) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); } } =head2 multiprocess_progressbar This function is used to display a progress bar during object scanning. =cut sub multiprocess_progressbar { my ($self) = @_; $self->logit("Starting progressbar writer process\n", 1); $0 = 'ora2pg logger'; $| = 1; my $DEBUG_PBAR = 0; my $width = 25; my $char = '='; my $kind = 'rows'; my $table_count = 0; my $table = ''; my $global_start_time = 0; my $total_rows = 0; my %table_progress = (); my $global_line_counter = 0; my $refresh_time = 3; #Update progress bar each 3 seconds my $last_refresh = time(); my $refresh_rows = 0; # Terminate the process when we doesn't read the complete file but must exit local $SIG{USR1} = sub { if ($global_line_counter) { my $end_time = time(); my $dt = $end_time - $global_start_time; $dt ||= 1; my $rps = int($global_line_counter / $dt); print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps tuples/sec)"), "\n"; } exit 0; }; $pipe->reader(); while ( my $r = <$pipe> ) { chomp($r); # When quit is received, then exit immediatly last if ($r eq 'quit'); # Store data export start time if ($r =~ /^GLOBAL EXPORT START TIME: (\d+)/) { print STDERR "GLOBAL EXPORT START TIME: $1\n" if ($DEBUG_PBAR); $global_start_time = $1; } # Store total number of tuples exported elsif ($r =~ /^GLOBAL EXPORT ROW NUMBER: (\d+)/) { print STDERR "GLOBAL EXPORT ROW NUMBER: $1\n" if ($DEBUG_PBAR); $total_rows = $1; } # A table export is starting (can be called multiple time with -J option) elsif ($r =~ /TABLE EXPORT IN PROGESS: (.*?), start: (\d+), rows (\d+)/) { print STDERR "TABLE EXPORT IN PROGESS: $1, start: $2, rows $3\n" if ($DEBUG_PBAR); $table_progress{$1}{start} = $2 if (!exists $table_progress{$1}{start}); $table_progress{$1}{rows} = $3 if (!exists $table_progress{$1}{rows}); } # A table export is ending elsif ($r =~ /TABLE EXPORT ENDED: (.*?), end: (\d+), rows (\d+)/) { print STDERR "TABLE EXPORT ENDED: $1, end: $2, rows $3\n" if ($DEBUG_PBAR); # Store timestamp at end of table export $table_progress{$1}{end} = $2; # Stores total number of rows exported when we do not used chunk of data if (!exists $table_progress{$1}{progress}) { $table_progress{$1}{progress} = $3; $global_line_counter += $3; } # Display table progression my $dt = $table_progress{$1}{end} - $table_progress{$1}{start}; my $rps = int($table_progress{$1}{progress}/ ($dt||1)); print STDERR $self->progress_bar($table_progress{$1}{progress}, $table_progress{$1}{rows}, 25, '=', 'rows', "Table $1 ($dt sec., $rps recs/sec)"), "\n"; # Display global export progression my $cur_time = time(); $dt = $cur_time - $global_start_time; $rps = int($global_line_counter/ ($dt || 1)); print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec), $1 in progress."), "\r"; $last_refresh = $cur_time; } # A chunk of DATA_LIMIT row is exported elsif ($r =~ /CHUNK \d+ DUMPED: (.*?), time: (\d+), rows (\d+)/) { print STDERR "CHUNK X DUMPED: $1, time: $2, rows $3\n" if ($DEBUG_PBAR); $table_progress{$1}{progress} += $3; $global_line_counter += $3; my $cur_time = time(); if ($cur_time >= ($last_refresh + $refresh_time)) { my $dt = $cur_time - $global_start_time; my $rps = int($global_line_counter/ ($dt || 1)); print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec), $1 in progress."), "\r"; $last_refresh = $cur_time; } } # A table export is ending elsif ($r =~ /TABLE EXPORT ENDED: (.*?), end: (\d+), report all parts/) { print STDERR "TABLE EXPORT ENDED: $1, end: $2, report all parts\n" if ($DEBUG_PBAR); # Store timestamp at end of table export $table_progress{$1}{end} = $2; # Get all statistics from multiple Oracle query for (my $i = 0; $i < $self->{oracle_copies}; $i++) { $table_progress{$1}{start} = $table_progress{"$1-part-$i"}{start} if (!exists $table_progress{$1}{start}); $table_progress{$1}{rows} = $table_progress{"$1-part-$i"}{rows}; delete $table_progress{"$1-part-$i"}; } # Stores total number of rows exported when we do not used chunk of data if (!exists $table_progress{$1}{progress}) { $table_progress{$1}{progress} = $3; $global_line_counter += $3; } # Display table progression my $dt = $table_progress{$1}{end} - $table_progress{$1}{start}; my $rps = int($table_progress{$1}{rows}/ ($dt||1)); print STDERR $self->progress_bar($table_progress{$1}{rows}, $table_progress{$1}{rows}, 25, '=', 'rows', "Table $1 ($dt sec., $rps recs/sec)"), "\n"; } else { print "PROGRESS BAR ERROR (unrecognized line sent to pipe): $r\n"; } } if ($global_line_counter) { my $end_time = time(); my $dt = $end_time - $global_start_time; $dt ||= 1; my $rps = int($global_line_counter / $dt); print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps tuples/sec)"), "\n"; } exit 0; } =head2 progress_bar This function is used to display a progress bar during object scanning. =cut sub progress_bar { my ($self, $got, $total, $width, $char, $kind, $msg) = @_; $width ||= 25; $char ||= '='; $kind ||= 'rows'; my $num_width = length $total; my $ratio = 1; if ($total > 0) { $ratio = $got / +$total; } my $len = (($width - 1) * $ratio); $len = $width - 1 if ($len >= $width); my $str = sprintf( "[%-${width}s] %${num_width}s/%s $kind (%.1f%%) $msg", $char x $len . '>', $got, $total, 100 * $ratio ); $len = length($str); $self->{prgb_len} ||= $len; if ($len < $self->{prgb_len}) { $str .= ' ' x ($self->{prgb_len} - $len); } $self->{prgb_len} = $len; # prepend time $str = '[' . strftime("%Y-%m-%d %H:%M:%S", localtime(time)) . '] ' . $str; return $str; } # Construct a query to exclude or only include some object wanted by the user # following the ALLOW and EXCLUDE configuration directive. The filter returned # must be used with the bind parameters stored in the @{$self->{query_bind_params}} # when calling the execute() function after the call of prepare(). sub limit_to_objects { my ($self, $obj_type, $column) = @_; my $str = ''; $obj_type ||= $self->{type}; $column ||= 'TABLE_NAME'; my @cols = split(/\|/, $column); my @arr_type = split(/\|/, $obj_type); my @done = (); my $has_limitation = 0; $self->{query_bind_params} = (); for (my $i = 0; $i <= $#arr_type; $i++) { my $colname = $cols[0]; $colname = $cols[$i] if (($#cols >= $i) && $cols[$i]); # Do not double exclusion/inclusion when column name is the same next if (grep(/^$colname$/, @done) && ! exists $self->{limited}{$arr_type[$i]}); push(@done, $colname); my $have_lookahead = 0; if ($#{$self->{limited}{$arr_type[$i]}} >= 0) { $str .= ' AND ('; if ($self->{db_version} =~ /Release [89]/) { for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { if ($self->{limited}{$arr_type[$i]}->[$j] =~ /^\!/) { $have_lookahead = 1; next; } $str .= "upper($colname) LIKE ?"; push(@{$self->{query_bind_params}}, uc($self->{limited}{$arr_type[$i]}->[$j])); if ($j < $#{$self->{limited}{$arr_type[$i]}}) { $str .= " OR "; } } $str =~ s/ OR $//; } else { for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { if ($self->{limited}{$arr_type[$i]}->[$j] =~ /^\!/) { $have_lookahead = 1; next; } if ($self->{is_mysql}) { $str .= "upper($colname) RLIKE ?" ; } elsif ($self->{is_mssql}) { #$str .= "PATINDEX(?, upper($colname)) != 0" ; $str .= "upper($colname) LIKE ?" ; } else { $str .= "REGEXP_LIKE(upper($colname), ?)" ; } my $objname = $self->{limited}{$arr_type[$i]}->[$j]; $objname =~ s/\$/\\\$/g; # support dollar sign if (!$self->{is_mssql}) { push(@{$self->{query_bind_params}}, uc("\^$objname\$")); } else { push(@{$self->{query_bind_params}}, uc("$objname")); } if ($j < $#{$self->{limited}{$arr_type[$i]}}) { $str .= " OR "; } } $str =~ s/ OR $//; } $str .= ')'; $str =~ s/ AND \(\)//; if ($have_lookahead) { if ($self->{db_version} =~ /Release [89]/) { for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { next if ($self->{limited}{$arr_type[$i]}->[$j] !~ /^\!(.+)/); $str .= " AND upper($colname) NOT LIKE ?"; push(@{$self->{query_bind_params}}, uc($1)); } } else { for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { next if ($self->{limited}{$arr_type[$i]}->[$j] !~ /^\!(.+)/); if ($self->{is_mysql}) { $str .= " AND upper($colname) NOT RLIKE ?" ; } elsif ($self->{is_mssql}) { #$str .= "PATINDEX(?, upper($colname)) != 0" ; $str .= "upper($colname) LIKE ?" ; } else { $str .= " AND NOT REGEXP_LIKE(upper($colname), ?)" ; } my $objname = $1; $objname =~ s/\$/\\\$/g; # support dollar sign if (!$self->{is_mssql}) { push(@{$self->{query_bind_params}}, uc("\^$objname\$")); } else { push(@{$self->{query_bind_params}}, uc("$objname")); } } } } $has_limitation = 1; } elsif ($#{$self->{excluded}{$arr_type[$i]}} >= 0) { if ($self->{db_version} =~ /Release [89]/) { $str .= ' AND ('; for (my $j = 0; $j <= $#{$self->{excluded}{$arr_type[$i]}}; $j++) { $str .= "upper($colname) NOT LIKE ?" ; push(@{$self->{query_bind_params}}, uc($self->{excluded}{$arr_type[$i]}->[$j])); if ($j < $#{$self->{excluded}{$arr_type[$i]}}) { $str .= " AND "; } } $str .= ')'; } else { $str .= ' AND ('; for (my $j = 0; $j <= $#{$self->{excluded}{$arr_type[$i]}}; $j++) { if ($self->{is_mysql}) { $str .= "upper($colname) NOT RLIKE ?" ; } elsif ($self->{is_mssql}) { #$str .= "PATINDEX(?, upper($colname)) = 0" ; $str .= "upper($colname) NOT LIKE ?" ; } else { $str .= "NOT REGEXP_LIKE(upper($colname), ?)" ; } if (!$self->{is_mssql}) { push(@{$self->{query_bind_params}}, uc("\^$self->{excluded}{$arr_type[$i]}->[$j]\$")); } else { push(@{$self->{query_bind_params}}, uc("$self->{excluded}{$arr_type[$i]}->[$j]")); } if ($j < $#{$self->{excluded}{$arr_type[$i]}}) { $str .= " AND "; } } $str .= ')'; } } # Always exclude unwanted tables if (!$self->{is_mysql} && !$self->{is_mssql} && !$self->{no_excluded_table} && !$has_limitation && ($arr_type[$i] =~ /TABLE|SEQUENCE|VIEW|TRIGGER|TYPE|SYNONYM/)) { if ($self->{db_version} =~ /Release [89]/) { $str .= ' AND ('; foreach my $t (@EXCLUDED_TABLES_8I) { $str .= " AND upper($colname) NOT LIKE ?"; push(@{$self->{query_bind_params}}, uc($t)); } $str .= ')'; } else { $str .= ' AND ( '; for (my $j = 0; $j <= $#EXCLUDED_TABLES; $j++) { if ($self->{is_mssql}) { #$str .= "PATINDEX(?, upper($colname)) = 0" ; $str .= "upper($colname) NOT LIKE ?" ; push(@{$self->{query_bind_params}}, uc("$EXCLUDED_TABLES[$j]")); } else { $str .= " NOT REGEXP_LIKE(upper($colname), ?)" ; push(@{$self->{query_bind_params}}, uc("\^$EXCLUDED_TABLES[$j]\$")); } if ($j < $#EXCLUDED_TABLES) { $str .= " AND "; } } $str .= ')'; } } } $str =~ s/ AND \( AND/ AND \(/g; $str =~ s/ AND \(\)//g; $str =~ s/ OR \(\)//g; return uc($str); } # Preload the bytea array at lib init BEGIN { build_escape_bytea(); } =head2 _count_check_constraint This function return the number of check constraints on a given table excluding CHECK IS NOT NULL constraint. =cut sub _count_check_constraint { my ($self, $check_constraint) = @_; my $num_chk_constr = 0; # Set the check constraint definition foreach my $k (keys %{$check_constraint->{constraint}}) { my $chkconstraint = $check_constraint->{constraint}->{$k}{condition}; next if (!$chkconstraint); # Skip NOT NULL constraint only next if ($chkconstraint =~ /^[^\s]+\s+IS\s+NOT\s+NULL$/i); $num_chk_constr++; } return $num_chk_constr; } =head2 _lookup_package This function is used to look at Oracle PACKAGE code to estimate the cost of a migration. It return an hash: function name => function code =cut sub _lookup_package { my ($self, $plsql) = @_; my $content = ''; my %infos = (); if ($plsql =~ /(?:CREATE|CREATE OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*PACKAGE\s+BODY\s*([^\s\%\(]+)((?:\s*\%ORA2PG_COMMENT\d+\%)*\s*(?:AS|IS))\s*(.*)/is) { my $pname = $1; my $type = $2; $content = $3; $pname =~ s/"//g; $self->logit("Looking at package $pname...\n", 1); $content =~ s/\bEND[^;]*;$//is; my @functions = $self->_extract_functions($content); foreach my $f (@functions) { next if (!$f); my %fct_detail = $self->_lookup_function($f, $pname); next if (!exists $fct_detail{name} || $self->excluded_functions($fct_detail{name})); $fct_detail{name} =~ s/^.*\.//; $fct_detail{name} =~ s/"//g; %{$infos{"$pname.$fct_detail{name}"}} = %fct_detail; } } return %infos; } # Returns 1 if the function match a EXCLUDED clause, 0 otherwise sub excluded_functions { my ($self, $fct_name) = @_; my @done = (); # Case where there is nothing to do here return 0 if (!$fct_name || (!exists $self->{excluded}{FUNCTION} && !exists $self->{excluded}{PROCEDURE})); push(@done, $fct_name); foreach my $type ('FUNCTION', 'PROCEDURE') { for (my $j = 0; $j <= $#{$self->{excluded}{$type}}; $j++) { if ($self->{excluded}{$type}->[$j] =~ /^!$fct_name$/i) { return 0; } elsif ($self->{excluded}{$type}->[$j] =~ /^$fct_name$/i) { return 1; } } } return 0; } =head2 _lookup_function This function is used to look at Oracle FUNCTION code to extract all parts of a fonction Return a hast with the details of the function =cut sub _lookup_function { my ($self, $plsql, $pname, $meta) = @_; if ($self->{is_mysql}) { return Ora2Pg::MySQL::_lookup_function($self, $plsql, $pname, $meta); } elsif ($self->{is_mssql}) { return Ora2Pg::MSSQL::_lookup_function($self, $plsql, $pname, $meta); } else { return Ora2Pg::Oracle::_lookup_function($self, $plsql, $pname, $meta); } } #### # Return a string to set the current search path #### sub set_search_path { my ($self, $owner, $pkg_path) = @_; my $local_path = ''; if ($self->{postgis_schema}) { $local_path = ',' . $self->quote_object_name($self->{postgis_schema}); } if ($self->{data_type}{BFILE} eq 'efile') { $local_path .= ',external_file'; } my $orafce_path = ''; $orafce_path = ',oracle' if ($self->{'use_orafce'}); $local_path .= "$orafce_path,public"; my $search_path = ''; if (!$self->{schema} && $self->{export_schema} && $owner) { $pkg_path = ',' . $pkg_path if ($pkg_path); $search_path = "SET search_path = " . $self->quote_object_name($owner) . "$pkg_path$local_path;"; } elsif (!$owner) { my @pathes = (); # When PG_SCHEMA is set, always take the value as search path if ($self->{pg_schema}) { @pathes = split(/\s*,\s*/, $self->{pg_schema}); } elsif ($self->{export_schema} && $self->{schema}) { # When EXPORT_SCHEMA is enable and we are working on a specific schema # set it as default search_path. Useful when object are not prefixed # with their destination schema. push(@pathes, $self->{schema}); } push(@pathes, $pkg_path) if ($pkg_path); if ($#pathes >= 0) { map { $_ = $self->quote_object_name($_); } @pathes; $search_path = "SET search_path = " . join(',', @pathes) . "$local_path;"; } } return "$search_path\n" if ($search_path); } sub _get_human_cost { my ($self, $total_cost_value) = @_; return 0 if (!$total_cost_value); my $human_cost = $total_cost_value * $self->{cost_unit_value}; if ($human_cost >= 420) { my $tmp = $human_cost/420; $tmp++ if ($tmp =~ s/\.\d+//); $human_cost = "$tmp person-day(s)"; } else { #my $tmp = $human_cost/60; #$tmp++ if ($tmp =~ s/\.\d+//); #$human_cost = "$tmp man-hour(s)"; # mimimum to 1 day, hours are not really relevant $human_cost = "1 person-day(s)"; } return $human_cost; } sub difficulty_assessment { my ($self, %report_info) = @_; # Migration that might be run automatically # 1 = trivial: no stored functions and no triggers # 2 = easy: no stored functions but with triggers # 3 = simple: stored functions and/or triggers # Migration that need code rewrite # 4 = manual: no stored functions but with triggers or view # 5 = difficult: with stored functions and/or triggers my $difficulty = 1; my @stored_function = ( 'FUNCTION', 'PACKAGE BODY', 'PROCEDURE' ); foreach my $n (@stored_function) { if (exists $report_info{'Objects'}{$n} && $report_info{'Objects'}{$n}{'number'}) { $difficulty = 3; last; } } if ($difficulty < 3) { $difficulty += 1 if ( exists $report_info{'Objects'}{'TRIGGER'} && $report_info{'Objects'}{'TRIGGER'}{'number'}); } if ($difficulty < 3) { foreach my $fct (keys %{ $report_info{'full_trigger_details'} } ) { next if (!exists $report_info{'full_trigger_details'}{$fct}{keywords}); $difficulty = 4; last; } } if ($difficulty <= 3) { foreach my $fct (keys %{ $report_info{'full_view_details'} } ) { next if (!exists $report_info{'full_view_details'}{$fct}{keywords}); $difficulty = 4; last; } } if ($difficulty >= 3) { foreach my $fct (keys %{ $report_info{'full_function_details'} } ) { next if (!exists $report_info{'full_function_details'}{$fct}{keywords}); $difficulty = 5; last; } foreach my $fct (keys %{ $report_info{'full_procedure_details'} } ) { next if (!exists $report_info{'full_procedure_details'}{$fct}{keywords}); $difficulty = 5; last; } foreach my $fct (keys %{ $report_info{'full_package_details'} } ) { next if (!exists $report_info{'full_package_details'}{$fct}{keywords}); $difficulty = 5; last; } } my $tmp = $report_info{'total_cost_value'}/84; $tmp++ if ($tmp =~ s/\.\d+//); my $level = 'A'; $level = 'B' if ($difficulty > 3); $level = 'C' if ( ($difficulty > 3) && ($tmp > $self->{human_days_limit}) ); return "$level-$difficulty"; } sub _show_report { my ($self, %report_info) = @_; my @ora_object_type = ( 'DATABASE LINK', 'DIRECTORY', 'FUNCTION', 'INDEX', 'JOB', 'MATERIALIZED VIEW', 'PACKAGE BODY', 'PROCEDURE', 'QUERY', 'SEQUENCE', 'SYNONYM', 'TABLE', 'TABLE PARTITION', 'TABLE SUBPARTITION', 'TRIGGER', 'TYPE', 'VIEW', # Other object type #CLUSTER #CONSUMER GROUP #DESTINATION #DIMENSION #EDITION #EVALUATION CONTEXT #INDEX PARTITION #INDEXTYPE #JAVA CLASS #JAVA DATA #JAVA RESOURCE #JAVA SOURCE #JOB CLASS #LIBRARY #LOB #LOB PARTITION #OPERATOR #PACKAGE #PROGRAM #QUEUE #RESOURCE PLAN #RULE #RULE SET #SCHEDULE #SCHEDULER GROUP #TYPE BODY #UNDEFINED #UNIFIED AUDIT POLICY #WINDOW #XML SCHEMA ); my $report_exported = 0; my $difficulty = $self->difficulty_assessment(%report_info); my $lbl_mig_type = qq{ Migration levels: A - Migration that might be run automatically B - Migration with code rewrite and a human-days cost up to $self->{human_days_limit} days C - Migration with code rewrite and a human-days cost above $self->{human_days_limit} days Technical levels: 1 = trivial: no stored functions and no triggers 2 = easy: no stored functions but with triggers, no manual rewriting 3 = simple: stored functions and/or triggers, no manual rewriting 4 = manual: no stored functions but with triggers or views with code rewriting 5 = difficult: stored functions and/or triggers with code rewriting }; # Generate report text report if (!$self->{dump_as_html} && !$self->{dump_as_csv} && !$self->{dump_as_sheet} && !$self->{dump_as_json}) { my $cost_header = ''; $report_exported = 1; $self->_select_output_file_suffix("txt"); $cost_header = "\tEstimated cost" if ($self->{estimate_cost}); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep("Ora2Pg v$VERSION - Database Migration Report\n"); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep("Version\t$report_info{'Version'}\n"); $self->logrep("Schema\t$report_info{'Schema'}\n"); $self->logrep("Size\t$report_info{'Size'}\n\n"); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep("Object\tNumber\tInvalid$cost_header\tComments\tDetails\n"); $self->logrep("-------------------------------------------------------------------------------\n"); foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/\. /gs; if ($self->{estimate_cost}) { $self->logrep("$typ\t$report_info{'Objects'}{$typ}{'number'}\t$report_info{'Objects'}{$typ}{'invalid'}\t$report_info{'Objects'}{$typ}{'cost_value'}\t$report_info{'Objects'}{$typ}{'comment'}\t$report_info{'Objects'}{$typ}{'detail'}\n"); } else { $self->logrep("$typ\t$report_info{'Objects'}{$typ}{'number'}\t$report_info{'Objects'}{$typ}{'invalid'}\t$report_info{'Objects'}{$typ}{'comment'}\t$report_info{'Objects'}{$typ}{'detail'}\n"); } } $self->logrep("-------------------------------------------------------------------------------\n"); if ($self->{estimate_cost}) { my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); my $comment = "$report_info{'total_cost_value'} cost migration units means approximatively $human_cost. The migration unit was set to $self->{cost_unit_value} minute(s)\n"; $self->logrep("Total\t$report_info{'total_object_number'}\t$report_info{'total_object_invalid'}\t$report_info{'total_cost_value'}\t$comment\n"); } else { $self->logrep("Total\t$report_info{'total_object_number'}\t$report_info{'total_object_invalid'}\n"); } $self->logrep("-------------------------------------------------------------------------------\n"); if ($self->{estimate_cost}) { $self->logrep("Migration level : $difficulty\n"); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep($lbl_mig_type); $self->logrep("-------------------------------------------------------------------------------\n"); if (scalar keys %{ $report_info{'full_function_details'} }) { $self->logrep("\nDetails of cost assessment per function\n"); foreach my $fct (sort { $report_info{'full_function_details'}{$b}{count} <=> $report_info{'full_function_details'}{$a}{count} } keys %{ $report_info{'full_function_details'} } ) { $self->logrep("Function $fct total estimated cost: $report_info{'full_function_details'}{$fct}{count}\n"); $self->logrep($report_info{'full_function_details'}{$fct}{info}); } $self->logrep("-------------------------------------------------------------------------------\n"); } if (scalar keys %{ $report_info{'full_procedure_details'} }) { $self->logrep("\nDetails of cost assessment per procedure\n"); foreach my $fct (sort { $report_info{'full_procedure_details'}{$b}{count} <=> $report_info{'full_procedure_details'}{$a}{count} } keys %{ $report_info{'full_procedure_details'} } ) { $self->logrep("Function $fct total estimated cost: $report_info{'full_procedure_details'}{$fct}{count}\n"); $self->logrep($report_info{'full_procedure_details'}{$fct}{info}); } $self->logrep("-------------------------------------------------------------------------------\n"); } if (scalar keys %{ $report_info{'full_package_details'} }) { $self->logrep("\nDetails of cost assessment per package function\n"); foreach my $fct (sort { $report_info{'full_package_details'}{$b}{count} <=> $report_info{'full_package_details'}{$a}{count} } keys %{ $report_info{'full_package_details'} } ) { $self->logrep("Function $fct total estimated cost: $report_info{'full_package_details'}{$fct}{count}\n"); $self->logrep($report_info{'full_package_details'}{$fct}{info}); } $self->logrep("-------------------------------------------------------------------------------\n"); } if (scalar keys %{ $report_info{'full_trigger_details'} }) { $self->logrep("\nDetails of cost assessment per trigger\n"); foreach my $fct (sort { $report_info{'full_trigger_details'}{$b}{count} <=> $report_info{'full_trigger_details'}{$a}{count} } keys %{ $report_info{'full_trigger_details'} } ) { $self->logrep("Trigger $fct total estimated cost: $report_info{'full_trigger_details'}{$fct}{count}\n"); $self->logrep($report_info{'full_trigger_details'}{$fct}{info}); } $self->logrep("-------------------------------------------------------------------------------\n"); } if (scalar keys %{ $report_info{'full_view_details'} }) { $self->logrep("\nDetails of cost assessment per view\n"); foreach my $fct (sort { $report_info{'full_view_details'}{$b}{count} <=> $report_info{'full_view_details'}{$a}{count} } keys %{ $report_info{'full_view_details'} } ) { $self->logrep("View $fct total estimated cost: $report_info{'full_view_details'}{$fct}{count}\n"); $self->logrep($report_info{'full_view_details'}{$fct}{info}); } $self->logrep("-------------------------------------------------------------------------------\n"); } } } if ($self->{dump_as_csv} && ($self->{dump_as_file_prefix} || !$report_exported)) { $report_exported = 1; $self->_select_output_file_suffix("csv"); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep("Ora2Pg v$VERSION - Database Migration Report\n"); $self->logrep("-------------------------------------------------------------------------------\n"); $self->logrep("Version\t$report_info{'Version'}\n"); $self->logrep("Schema\t$report_info{'Schema'}\n"); $self->logrep("Size\t$report_info{'Size'}\n\n"); $self->logrep("-------------------------------------------------------------------------------\n\n"); $self->logrep("Object;Number;Invalid;Estimated cost;Comments\n"); foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/\. /gs; $self->logrep("$typ;$report_info{'Objects'}{$typ}{'number'};$report_info{'Objects'}{$typ}{'invalid'};$report_info{'Objects'}{$typ}{'cost_value'};$report_info{'Objects'}{$typ}{'comment'}\n"); } my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); $difficulty = '' if (!$self->{estimate_cost}); $self->logrep("\n"); $self->logrep("Total Number;Total Invalid;Total Estimated cost;Human days cost;Migration level\n"); $self->logrep("$report_info{'total_object_number'};$report_info{'total_object_invalid'};$report_info{'total_cost_value'};$human_cost;$difficulty\n"); } if ($self->{dump_as_json} && ($self->{dump_as_file_prefix} || !$report_exported)) { $report_exported = 1; $self->_select_output_file_suffix("json"); $self->logrep("{\n"); $self->logrep("\"ora2pg version\": $VERSION,\n"); $self->logrep("\"Version\": \"$report_info{'Version'}\",\n"); $self->logrep("\"Schema\": \"$report_info{'Schema'}\",\n"); $self->logrep("\"Size\": \"$report_info{'Size'}\",\n"); my $cnt=0; $self->logrep("\"objects\": ["); foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/\. /gs; $cnt++; if ($cnt ne 1) { $self->logrep(","); } $self->logrep("{"); $self->logrep("\"object\":\"$typ\",\"number\":$report_info{'Objects'}{$typ}{'number'},"); $self->logrep("\"invalid\":$report_info{'Objects'}{$typ}{'invalid'},"); $self->logrep("\"cost value\":$report_info{'Objects'}{$typ}{'cost_value'},"); my $json_comment = ($report_info{'Objects'}{$typ}{'comment'} =~ s/\n/\\n/gr); $self->logrep("\"comment\":\"$json_comment\",\n"); $self->logrep("\"details\":\"$report_info{'Objects'}{$typ}{'detail'}\"}\n"); } $self->logrep("]\n"); my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); $difficulty = '' if (!$self->{estimate_cost}); $self->logrep(",\"total number\":$report_info{'total_object_number'}"); $self->logrep(",\"total invalid\":$report_info{'total_object_invalid'}"); $self->logrep(",\"total cost\":$report_info{'total_cost_value'}"); $self->logrep(",\"human days cost\":\"$human_cost\""); $self->logrep(",\"migration level\":\"$difficulty\""); $self->logrep("}\n"); } if ($self->{dump_as_sheet} && ($self->{dump_as_file_prefix} || !$report_exported)) { $report_exported = 1; $self->_select_output_file_suffix("sheet.csv"); $difficulty = '' if (!$self->{estimate_cost}); my @header = ('Instance', 'Version', 'Schema', 'Size', 'Cost assessment', 'Migration type'); my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); my @infos = ($self->{oracle_dsn}, $report_info{'Version'}, $report_info{'Schema'}, $report_info{'Size'}, $human_cost, $difficulty); foreach my $typ (sort @ora_object_type) { push(@header, $typ); $report_info{'Objects'}{$typ}{'number'} ||= 0; $report_info{'Objects'}{$typ}{'invalid'} ||= 0; $report_info{'Objects'}{$typ}{'cost_value'} ||= 0; push(@infos, "$report_info{'Objects'}{$typ}{'number'}/$report_info{'Objects'}{$typ}{'invalid'}/$report_info{'Objects'}{$typ}{'cost_value'}"); } push(@header, "Total assessment"); push(@infos, "$report_info{total_object_number}/$report_info{total_object_invalid}/$report_info{total_cost_value}"); if ($self->{print_header}) { $self->logrep('"' . join('";"', @header) . '"' . "\n"); } $self->logrep('"' . join('";"', @infos) . '"' . "\n"); } if ($self->{dump_as_html} && ($self->{dump_as_file_prefix} || !$report_exported)) { $report_exported = 1; $self->_select_output_file_suffix("html"); my $cost_header = ''; $cost_header = "Estimated cost" if ($self->{estimate_cost}); my $date = localtime(time); my $html_header = qq{ Ora2Pg - Database Migration Report
$cost_header }; $self->logrep($html_header); foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/
/gs; $report_info{'Objects'}{$typ}{'detail'} = "
See details$report_info{'Objects'}{$typ}{'detail'}
" if ($report_info{'Objects'}{$typ}{'detail'} ne ''); if ($self->{estimate_cost}) { $self->logrep("\n"); } else { $self->logrep("\n"); } } if ($self->{estimate_cost}) { my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); my $comment = "$report_info{'total_cost_value'} cost migration units means approximatively $human_cost. The migration unit was set to $self->{cost_unit_value} minute(s)\n"; $self->logrep("\n"); } else { $self->logrep("\n"); } $self->logrep("
ObjectNumberInvalidCommentsDetails
$typ$report_info{'Objects'}{$typ}{'number'}$report_info{'Objects'}{$typ}{'invalid'}$report_info{'Objects'}{$typ}{'cost_value'}$report_info{'Objects'}{$typ}{'comment'}$report_info{'Objects'}{$typ}{'detail'}
$typ$report_info{'Objects'}{$typ}{'number'}$report_info{'Objects'}{$typ}{'invalid'}$report_info{'Objects'}{$typ}{'comment'}$report_info{'Objects'}{$typ}{'detail'}
Total$report_info{'total_object_number'}$report_info{'total_object_invalid'}$report_info{'total_cost_value'}$comment
Total$report_info{'total_object_number'}$report_info{'total_object_invalid'}
\n
\n"); if ($self->{estimate_cost}) { $self->logrep("

Migration level: $difficulty

\n"); $lbl_mig_type = qq{
  • Migration levels:
    • A - Migration that might be run automatically
    • B - Migration with code rewrite and a human-days cost up to $self->{human_days_limit} days
    • C - Migration with code rewrite and a human-days cost above $self->{human_days_limit} days
  • Technical levels:
    • 1 = trivial: no stored functions and no triggers
    • 2 = easy: no stored functions but with triggers, no manual rewriting
    • 3 = simple: stored functions and/or triggers, no manual rewriting
    • 4 = manual: no stored functions but with triggers or views with code rewriting
    • 5 = difficult: stored functions and/or triggers with code rewriting
}; $self->logrep($lbl_mig_type); if (scalar keys %{ $report_info{'full_function_details'} }) { $self->logrep("

Details of cost assessment per function

\n"); $self->logrep("
Show
    \n"); foreach my $fct (sort { $report_info{'full_function_details'}{$b}{count} <=> $report_info{'full_function_details'}{$a}{count} } keys %{ $report_info{'full_function_details'} } ) { $self->logrep("
  • Function $fct total estimated cost: $report_info{'full_function_details'}{$fct}{count}
    • \n"); $self->logrep("
      \n"); $report_info{'full_function_details'}{$fct}{info} =~ s/\t/
    • /gs; $report_info{'full_function_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; $self->logrep($report_info{'full_function_details'}{$fct}{info}); $self->logrep("
    \n"); } $self->logrep("
\n"); } if (scalar keys %{ $report_info{'full_procedure_details'} }) { $self->logrep("

Details of cost assessment per procedure

\n"); $self->logrep("
Show
    \n"); foreach my $fct (sort { $report_info{'full_procedure_details'}{$b}{count} <=> $report_info{'full_procedure_details'}{$a}{count} } keys %{ $report_info{'full_procedure_details'} } ) { $self->logrep("
  • Procedure $fct total estimated cost: $report_info{'full_procedure_details'}{$fct}{count}
    • \n"); $self->logrep("
      \n"); $report_info{'full_procedure_details'}{$fct}{info} =~ s/\t/
    • /gs; $report_info{'full_procedure_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; $self->logrep($report_info{'full_procedure_details'}{$fct}{info}); $self->logrep("
    \n"); } $self->logrep("
\n"); } if (scalar keys %{ $report_info{'full_package_details'} }) { $self->logrep("

Details of cost assessment per package function

\n"); $self->logrep("
Show
    \n"); foreach my $fct (sort { $report_info{'full_package_details'}{$b}{count} <=> $report_info{'full_package_details'}{$a}{count} } keys %{ $report_info{'full_package_details'} } ) { $self->logrep("
  • Function $fct total estimated cost: $report_info{'full_package_details'}{$fct}{count}
    • \n"); $self->logrep("
      \n"); $report_info{'full_package_details'}{$fct}{info} =~ s/\t/
    • /gs; $report_info{'full_package_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; $self->logrep($report_info{'full_package_details'}{$fct}{info}); $self->logrep("
    \n"); } $self->logrep("
\n"); } if (scalar keys %{ $report_info{'full_trigger_details'} }) { $self->logrep("

Details of cost assessment per trigger

\n"); $self->logrep("
Show
    \n"); foreach my $fct (sort { $report_info{'full_trigger_details'}{$b}{count} <=> $report_info{'full_trigger_details'}{$a}{count} } keys %{ $report_info{'full_trigger_details'} } ) { $self->logrep("
  • Trigger $fct total estimated cost: $report_info{'full_trigger_details'}{$fct}{count}
    • \n"); $self->logrep("
      \n"); $report_info{'full_trigger_details'}{$fct}{info} =~ s/\t/
    • /gs; $report_info{'full_trigger_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; $self->logrep($report_info{'full_trigger_details'}{$fct}{info}); $self->logrep("
    \n"); } $self->logrep("
\n"); } if (scalar keys %{ $report_info{'full_view_details'} }) { $self->logrep("

Details of cost assessment per view

\n"); $self->logrep("
Show
    \n"); foreach my $fct (sort { $report_info{'full_view_details'}{$b}{count} <=> $report_info{'full_view_details'}{$a}{count} } keys %{ $report_info{'full_view_details'} } ) { $self->logrep("
  • View $fct total estimated cost: $report_info{'full_view_details'}{$fct}{count}
    • \n"); $self->logrep("
      \n"); $report_info{'full_view_details'}{$fct}{info} =~ s/\t/
    • /gs; $report_info{'full_view_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; $self->logrep($report_info{'full_view_details'}{$fct}{info}); $self->logrep("
    \n"); } $self->logrep("
\n"); } } my $html_footer = qq{ }; $self->logrep($html_footer); } } sub get_kettle_xml { return < template Normal 0 / ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDTRANSNAMEYTRANSNAMESTATUSYSTATUSLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSSTARTDATEYSTARTDATEENDDATEYENDDATELOGDATEYLOGDATEDEPDATEYDEPDATEREPLAYDATEYREPLAYDATELOG_FIELDYLOG_FIELD
ID_BATCHYID_BATCHSEQ_NRYSEQ_NRLOGDATEYLOGDATETRANSNAMEYTRANSNAMESTEPNAMEYSTEPNAMESTEP_COPYYSTEP_COPYLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSINPUT_BUFFER_ROWSYINPUT_BUFFER_ROWSOUTPUT_BUFFER_ROWSYOUTPUT_BUFFER_ROWS
ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDLOG_DATEYLOG_DATELOGGING_OBJECT_TYPEYLOGGING_OBJECT_TYPEOBJECT_NAMEYOBJECT_NAMEOBJECT_COPYYOBJECT_COPYREPOSITORY_DIRECTORYYREPOSITORY_DIRECTORYFILENAMEYFILENAMEOBJECT_IDYOBJECT_IDOBJECT_REVISIONYOBJECT_REVISIONPARENT_CHANNEL_IDYPARENT_CHANNEL_IDROOT_CHANNEL_IDYROOT_CHANNEL_ID
ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDLOG_DATEYLOG_DATETRANSNAMEYTRANSNAMESTEPNAMEYSTEPNAMESTEP_COPYYSTEP_COPYLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSLOG_FIELDNLOG_FIELD
0.0 0.0 __rowset__ 10 10 N Y 500000 Y Y 1000 100 - 2013/02/28 14:04:49.560 - 2013/03/01 12:35:39.999 __oracle_db__ __oracle_host__ ORACLE Native __oracle_instance__ __oracle_port__ __oracle_username__ __oracle_password__ EXTRA_OPTION_ORACLE.defaultRowPrefetch10000 EXTRA_OPTION_ORACLE.fetchSize1000 FORCE_IDENTIFIERS_TO_LOWERCASEN FORCE_IDENTIFIERS_TO_UPPERCASEN IS_CLUSTEREDN PORT_NUMBER__oracle_port__ QUOTE_ALL_FIELDSN SUPPORTS_BOOLEAN_DATA_TYPEN USE_POOLINGN __postgres_db__ __postgres_host__ POSTGRESQL Native __postgres_database_name__ __postgres_port__ __postgres_username__ __postgres_password__ FORCE_IDENTIFIERS_TO_LOWERCASEN FORCE_IDENTIFIERS_TO_UPPERCASEN IS_CLUSTEREDN PORT_NUMBER__postgres_port__ QUOTE_ALL_FIELDSN SUPPORTS_BOOLEAN_DATA_TYPEY USE_POOLINGN EXTRA_OPTION_POSTGRESQL.synchronous_commit__sync_commit_onoff__ Table inputModified Java Script ValueY Modified Java Script ValueTable outputY Table input TableInput Y __select_copies__ none __oracle_db__ __select_query__ 0 N N N 122 160 Y Table output TableOutput Y __insert_copies__ none __postgres_db__
__postgres_table_name__
__commit_size__ __truncate__ Y Y N N N Y N Y N 369 155 Y Modified Java Script Value ScriptValueMod Y __js_copies__ none N 9 0 Script 1 for (var i=0;i<getInputRowMeta().size();i++) { var valueMeta = getInputRowMeta().getValueMeta(i); if (valueMeta.getTypeDesc().equals("String")) { row[i]=replace(row[i],"\\00",''); } } 243 166 Y N EOF } # Constants for creating kettle files from the template sub create_kettle_output { my ($self, $table, $output_dir) = @_; my $oracle_host = 'localhost'; if ($self->{oracle_dsn} =~ /host=([^;]+)/) { $oracle_host = $1; } my $oracle_port = 1521; if ($self->{oracle_dsn} =~ /port=(\d+)/) { $oracle_port = $1; } my $oracle_instance=''; if ($self->{oracle_dsn} =~ /sid=([^;]+)/) { $oracle_instance = $1; } elsif ($self->{oracle_dsn} =~ /dbi:Oracle:([^:]+)/) { $oracle_instance = $1; } if ($self->{oracle_dsn} =~ /\/\/([^:]+):(\d+)\/(.*)/) { $oracle_host = $1; $oracle_port = $2; $oracle_instance = $3; } elsif ($self->{oracle_dsn} =~ /\/\/([^\/]+)\/(.*)/) { $oracle_host = $1; $oracle_instance = $2; } my $pg_host = 'localhost'; if ($self->{pg_dsn} =~ /host=([^;]+)/) { $pg_host = $1; } my $pg_port = 5432; if ($self->{pg_dsn} =~ /port=(\d+)/) { $pg_port = $1; } my $pg_dbname = ''; if ($self->{pg_dsn} =~ /dbname=([^;]+)/) { $pg_dbname = $1; } my $select_query = "SELECT * FROM $table"; if ($self->{schema}) { $select_query = "SELECT * FROM $self->{schema}.$table"; } my $select_copies = $self->{oracle_copies} || 1; if (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) { my $colpk = $self->{defined_pk}{"\L$table\E"}; if ($self->{preserve_case}) { $colpk = '"' . $colpk . '"'; } if ($self->{schema}) { $select_query = "SELECT * FROM $self->{schema}.$table WHERE "; } else { $select_query = "SELECT * FROM $table WHERE "; } if ($self->{is_mssql}) { $select_query = "ABS($colpk % \${Internal.Step.Unique.Count})=\${Internal.Step.Unique.Number}"; } else { $select_query = "ABS(MOD($colpk,\${Internal.Step.Unique.Count}))=\${Internal.Step.Unique.Number}"; } } else { $select_copies = 1; } my $insert_copies = $self->{jobs} || 4; my $js_copies = $insert_copies; my $rowset = $self->{data_limit} || 10000; if (exists $self->{local_data_limit}{$table}) { $rowset = $self->{local_data_limit}{$table}; } my $commit_size = 500; my $sync_commit_onoff = 'off'; my $truncate = 'Y'; $truncate = 'N' if (!$self->{truncate_table}); my $pg_table = $table; if ($self->{export_schema}) { if ($self->{pg_schema}) { $pg_table = "$self->{pg_schema}.$table"; } elsif ($self->{schema}) { $pg_table = "$self->{schema}.$table"; } } my $xml = &get_kettle_xml(); $xml =~ s/__oracle_host__/$oracle_host/gs; $xml =~ s/__oracle_instance__/$oracle_instance/gs; $xml =~ s/__oracle_port__/$oracle_port/gs; $xml =~ s/__oracle_username__/$self->{oracle_user}/gs; $xml =~ s/__oracle_password__/$self->{oracle_pwd}/gs; $xml =~ s/__postgres_host__/$pg_host/gs; $xml =~ s/__postgres_database_name__/$pg_dbname/gs; $xml =~ s/__postgres_port__/$pg_port/gs; $xml =~ s/__postgres_username__/$self->{pg_user}/gs; $xml =~ s/__postgres_password__/$self->{pg_pwd}/gs; $xml =~ s/__select_copies__/$select_copies/gs; $xml =~ s/__select_query__/$select_query/gs; $xml =~ s/__insert_copies__/$insert_copies/gs; $xml =~ s/__js_copies__/$js_copies/gs; $xml =~ s/__truncate__/$truncate/gs; $xml =~ s/__transformation_name__/$table/gs; $xml =~ s/__postgres_table_name__/$pg_table/gs; $xml =~ s/__rowset__/$rowset/gs; $xml =~ s/__commit_size__/$commit_size/gs; $xml =~ s/__sync_commit_onoff__/$sync_commit_onoff/gs; my $fh = new IO::File; $fh->open(">$output_dir$table.ktr") or $self->logit("FATAL: can't write to $output_dir$table.ktr, $!\n", 0, 1); $fh->print($xml); $self->close_export_file($fh); return "JAVAMAXMEM=4096 ./pan.sh -file \$KETTLE_TEMPLATE_PATH/$output_dir$table.ktr -level Detailed\n"; } # Normalize SQL queries by removing parameters sub normalize_query { my ($self, $orig_query) = @_; return if (!$orig_query); # Remove comments $orig_query =~ s/\/\*(.*?)\*\///gs; # Set the entire query lowercase $orig_query = lc($orig_query); # Remove extra space, new line and tab characters by a single space $orig_query =~ s/\s+/ /gs; # Removed start of transaction if ($orig_query !~ /^\s*begin\s*;\s*$/) { $orig_query =~ s/^\s*begin\s*;\s*//gs } # Remove string content $orig_query =~ s/\\'//g; $orig_query =~ s/'[^']*'/''/g; $orig_query =~ s/''('')+/''/g; # Remove NULL parameters $orig_query =~ s/=\s*NULL/=''/g; # Remove numbers $orig_query =~ s/([^a-z_\$-])-?([0-9]+)/${1}0/g; # Remove hexadecimal numbers $orig_query =~ s/([^a-z_\$-])0x[0-9a-f]{1,10}/${1}0x/g; # Remove IN values $orig_query =~ s/in\s*\([\'0x,\s]*\)/in (...)/g; return $orig_query; } sub _escape_lob { my ($self, $col, $generic_type, $cond, $isnested, $dest_type) = @_; if ($self->{type} eq 'COPY') { if ($generic_type eq 'BLOB') { # Get an hexa representation of the blob data $col = unpack("H*",$col); $col = "\\\\x" . $col; } elsif ($generic_type eq 'RAW') { # RAW data are already returned in hexa by DBD::Oracle if ($dest_type eq 'uuid') { # we do nothing, the value will be cast into uuid automatically } elsif ($dest_type eq 'bytea' && $cond->{long}) { $col = unpack("H*",$col); $col = "\\\\x" . $col; } elsif ($dest_type eq 'bytea') { $col = "\\\\x" . $col; } } elsif (($generic_type eq 'CLOB') || $cond->{istext}) { $col = $self->escape_copy($col, $isnested); } } else { if ($generic_type eq 'BLOB') { # Get an hexa representation of the blob data $col = unpack("H*",$col); if (!$self->{standard_conforming_strings}) { $col = "'$col'"; } else { $col = "E'$col'"; } if (!$self->{blob_to_lo}) { if (!$self->{pg_dsn}) { $col = "decode($col, 'hex')"; } else { # with prepare just send the data $col =~ s/^[E]?'//; $col =~ s/'$//; } } elsif (!$self->{pg_dsn}) { $col = "lo_from_bytea(0, decode($col, 'hex'))"; } } elsif ($generic_type eq 'RAW') { # RAW data are already returned in hexa by DBD::Oracle if ($dest_type eq 'uuid') { # we do nothing, the value will be cast into uuid automatically $col = "'$col'"; } elsif ($dest_type eq 'bytea') { if (!$self->{standard_conforming_strings}) { $col = "'$col'"; } else { $col = "E'$col'"; } $col = "decode($col, 'hex')"; } } elsif (($generic_type eq 'CLOB') || $cond->{istext}) { $col = $self->escape_insert($col, $isnested); } } return $col; } sub escape_copy { my ($self, $col, $isnested) = @_; my $q = "'"; $q = '"' if ($isnested); if ($self->{has_utf8_fct}) { utf8::encode($col) if (!utf8::valid($col)); } # Escape some character for COPY output $col =~ s/(\0|\\|\r|\n|\t)/$ESCAPE_COPY->{$1}/gs; if (!$self->{noescape}) { $col =~ s/\f/\\f/gs; $col =~ s/([\1-\10\13-\14\16-\37])/sprintf("\\%03o", ord($1))/egs; } return $col; } sub escape_insert { my ($self, $col, $isnested) = @_; my $q = "'"; $q = '"' if ($isnested); if (!$self->{standard_conforming_strings}) { $col =~ s/'/''/gs; # double single quote if ($isnested) { $col =~ s/"/\\"/gs; # escape double quote } $col =~ s/\\/\\\\/gs; $col =~ s/\0//gs; $col = "$q$col$q"; } else { $col =~ s/\0//gs; $col =~ s/\\/\\\\/gs; $col =~ s/\r/\\r/gs; $col =~ s/\n/\\n/gs; if ($isnested) { $col =~ s/'/''/gs; # double single quote $col =~ s/"/\\"/gs; # escape double quote $col = "$q$col$q"; } else { $col =~ s/'/\\'/gs; # escape single quote $col = "E'$col'"; } } return $col; } sub clear_global_declaration { my ($self, $pname, $str, $is_pkg_body) = @_; # Remove comment $str =~ s/\%ORA2PG_COMMENT\d+\%//igs; # remove pragma restrict_references $str =~ s/PRAGMA\s+RESTRICT_REFERENCES\s*\([^;]+;//igs; # Remove all function/procedure declaration from the content if (!$is_pkg_body) { $str =~ s/\b(PROCEDURE|FUNCTION)\s+[^;]+;//igs; } else { while ($str =~ s/\b(PROCEDURE|FUNCTION)\s+.*?END[^;]*;((?:(?!\bEND\b).)*\s+(?:PROCEDURE|FUNCTION)\s+)/$2/is) { }; $str =~ s/(PROCEDURE|FUNCTION).*END[^;]*;//is; } # Remove end of the package declaration $str =~ s/\s+END[^;]*;\s*$//igs; # Eliminate extra newline $str =~ s/[\r\n]+/\n/isg; my @cursors = (); while ($str =~ s/(CURSOR\s+[^;]+\s+RETURN\s+[^;]+;)//is) { push(@cursors, $1); } # Extract TYPE/SUBTYPE declaration my $i = 0; while ($str =~ s/\b(SUBTYPE|TYPE)\s+([^\s\(\)]+)\s+(AS|IS)\s+([^;]+;)//is) { $self->{pkg_type}{$pname}{$2} = "$pname.$2"; my $code = "$1 $self->{pkg_type}{$pname}{$2} AS $4"; push(@{$self->{types}}, { ('name' => $2, 'code' => $code, 'pos' => $i++) }); } return ($str, @cursors); } sub register_global_variable { my ($self, $pname, $glob_vars) = @_; $glob_vars = $self->_replace_sql_type($glob_vars); # Replace PL/SQL code into PL/PGSQL similar code $glob_vars = Ora2Pg::PLSQL::convert_plsql_code($self, $glob_vars); my @vars = split(/\s*(\%ORA2PG_COMMENT\d+\%|;)\s*/, $glob_vars); map { s/^\s+//; s/\s+$//; } @vars; my $ret = ''; foreach my $l (@vars) { if ($l eq ';' || $l =~ /ORA2PG_COMMENT/ || $l =~ /^CREATE\s+/i) { $ret .= $l if ($l ne ';'); next; } next if (!$l); $l =~ s/\-\-[^\r\n]+//sg; $l =~ s/\s*:=\s*/ := /igs; my ($n, $type, @others) = split(/\s+/, $l); $ret .= $l, next if (!$type); if (!$n) { $n = $type; $type = $others[0] || ''; } if (uc($type) eq 'EXCEPTION') { $n = lc($n); if (!exists $self->{custom_exception}{$n}) { $self->{custom_exception}{$n} = $self->{exception_id}++; } next; } next if (!$pname); my $v = lc($pname . '.' . $n); $self->{global_variables}{$v}{name} = lc($n); if (uc($type) eq 'CONSTANT') { $type = ''; $self->{global_variables}{$v}{constant} = 1; for (my $j = 0; $j < $#others; $j++) { $type .= $others[$j] if ($others[$j] ne ':=' and uc($others[$j]) ne 'DEFAULT'); } } # extract the default value from the declaration for (my $j = 0; $j < $#others; $j++) { if ($others[$j] eq ':=' or uc($others[$j]) eq 'DEFAULT') { # Append the rest of the definition to the default value for (my $l = $j+1; $l <= $#others; $l++) { $self->{global_variables}{$v}{default} .= " " if ($l > $j+1); $self->{global_variables}{$v}{default} .= $others[$l]; } last; } } if (exists $self->{global_variables}{$v}{default}) { #$self->{global_variables}{$v}{default} =~ s/^\s+//; $self->{global_variables}{$v}{default} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{global_variables}{$v}{default}); $self->_restore_text_constant_part(\$self->{global_variables}{$v}{default}); if ($self->{global_variables}{$v}{default} !~ /^'/) { $self->{global_variables}{$v}{default} =~ s/'/\\'/gs; } $self->{global_variables}{$v}{default} =~ s/^'//s; $self->{global_variables}{$v}{default} =~ s/'\s*$//s; } $self->{global_variables}{$v}{type} = $type; # Handle Oracle user defined error code if ($self->{global_variables}{$v}{constant} && ($type =~ /bigint|int|numeric|double/) && $self->{global_variables}{$v}{default} <= -20000 && $self->{global_variables}{$v}{default} >= -20999) { $self->{global_variables}{$v}{default} =~ s/^\s+//; # Change the type into char(5) for SQLSTATE type $self->{global_variables}{$v}{type} = 'char(5)'; # Transform the value to match PostgreSQL user defined exceptions starting with 45 $self->{global_variables}{$v}{default} =~ s/^-20/45/; } } return $ret; } sub remove_newline { my $str = shift; $str =~ s/[\n\r]+\s*/ /gs; return $str; } sub _ask_username { my $self = shift; my $target = shift; print 'Enter ' . $target . ' username: '; my $username = ReadLine(0); chomp($username); return $username; } sub _ask_password { my $self = shift; my $target = shift; print 'Enter ' . $target . ' password: '; ReadMode(2); my $password = ReadLine(0); ReadMode(0); chomp($password); print "\n"; return $password; } ############## # Prefix function calls with their package name when necessary ############## sub normalize_function_call { my ($self, $str) = @_; return if (!$self->{current_package}); my $p = lc($self->{current_package}); # foreach function declared in a package qualify its callis with the package name foreach my $f (keys %{$self->{package_functions}{$p}}) { # If the package is already prefixed to the function name in the hash take it from here if (lc($self->{package_functions}{$p}{$f}{name}) ne lc($f)) { $$str =~ s/([^\.])\b$f\s*([\(;])/$1$self->{package_functions}{$p}{$f}{name}$2/igs; } elsif (exists $self->{package_functions}{$p}{$f}{package}) { # otherwise use the package name from the hash and the function name from the string $$str =~ s/([^\.])\b($f\s*[\(;])/$1$self->{package_functions}{$p}{$f}{package}\.$2/igs; } # Append parenthesis to functions without parameters $$str =~ s/\b($self->{package_functions}{$p}{$f}{package}\.$f)\b((?!\s*\())/$1()$2/igs; } # Fix unwanted double parenthesis #$$str =~ s/\(\)\s*(\()/ $1/gs; } ############## # Requalify function calls ############## sub requalify_function_call { my ($self, $str) = @_; # Loop through package foreach my $p (keys %{$self->{package_functions}}) { # foreach function declared in a package qualify its callis with the package name foreach my $f (keys %{$self->{package_functions}{$p}}) { $$str =~ s/\b$p\.$f\s*([\(;])/$self->{package_functions}{$p}{$f}{name}$1/igs; } } } sub _make_WITH { my ($with_oid, $table_info) = @_; my @withs =(); push @withs, 'OIDS' if ($with_oid); push @withs, 'fillfactor=' . $table_info->{fillfactor} if (exists $table_info->{fillfactor}); my $WITH=''; if (@withs>0) { $WITH .= 'WITH (' . join(",",@withs) . ')'; } return $WITH; } sub _create_foreign_server { my $self = shift; # Verify that the oracle_fdw or mysql_fdw extension is created, create it if not my $sth = $self->{dbhdest}->prepare("SELECT * FROM pg_extension WHERE extname=?") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); my $extension = 'oracle_fdw'; $extension = 'mysql_fdw' if ($self->{is_mysql}); $extension = 'tds_fdw' if ($self->{is_mssql}); $sth->execute($extension) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . ", SQL: SELECT * FROM pg_extension WHERE extname='$extension'\n", 0, 1); my $row = $sth->fetch; $sth->finish; if (not defined $row) { # try to create the extension $self->{dbhdest}->do("CREATE EXTENSION IF NOT EXISTS $extension") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } # Check if the server already exists or need to be created $sth = $self->{dbhdest}->prepare("SELECT * FROM pg_foreign_server WHERE srvname=?") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); $sth->execute($self->{fdw_server}) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); $row = $sth->fetch; $sth->finish; if (not defined $row) { # try to create the foreign server if (!defined $self->{oracle_pwd}) { eval("use Term::ReadKey;") unless $self->{oracle_user} eq '/'; if (!$@) { $self->{oracle_user} = $self->_ask_username('Oracle') unless (defined $self->{oracle_user}); $self->{oracle_pwd} = $self->_ask_password('Oracle') unless ($self->{oracle_user} eq '/'); } } my $ora_session_mode = ($self->{oracle_user} eq "/" || $self->{oracle_user} eq "sys") ? 2 : undef; $self->logit("ORACLE_HOME = $ENV{ORACLE_HOME}\n", 1); $self->logit("NLS_LANG = $ENV{NLS_LANG}\n", 1); $self->logit("NLS_NCHAR = $ENV{NLS_NCHAR}\n", 1); $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$quiet); my $sql = ''; my $extension = 'oracle_fdw'; if (!$self->{fdw_server}) { $self->logit("FATAL: a foreign server name must be set using FDW_SERVER\n", 0, 1); } if (!$self->{is_mysql} && !$self->{is_mssql} && ($self->{oracle_dsn} =~ /(\/\/.*\/.*)/ || $self->{oracle_dsn} =~ /dbi:Oracle:([^=:;]+)$/i) ) { $self->{oracle_fwd_dsn} = "dbserver '$1'"; } else { $self->{oracle_dsn} =~ /(?:host|server)=([^;]+)/; my $host = $1 || 'localhost'; $self->{oracle_dsn} =~ /port=(\d+)/; my $port = $1; if (!$port && $self->{is_mysql}) { $port = 3306; } elsif (!$port && $self->{is_mssql}) { $port = 1433; } elsif (!$port) { $port = 1521; } my $sid = ''; if ($self->{is_mysql}) { $extension = 'mysql_fdw'; $self->{oracle_dsn} =~ /(database)=([^;]+)/; $self->{mysql_fwd_db} = $2 || ''; $self->{oracle_fwd_dsn} = "host '$host', port '$port'"; } elsif ($self->{is_mssql}) { $extension = 'tds_fdw'; $self->{oracle_dsn} =~ /(database)=([^;]+)/; $self->{mssql_fwd_db} = $2 || ''; $self->{oracle_fwd_dsn} = "servername '$host', port '$port', database '$self->{mssql_fwd_db}'"; } else { $self->{oracle_dsn} =~ /(service_name|sid)=([^;]+)/; $sid = $2 || ''; $self->{oracle_fwd_dsn} = "dbserver '//$host:$port/$sid'"; } } $sql = "CREATE SERVER $self->{fdw_server} FOREIGN DATA WRAPPER $extension OPTIONS ($self->{oracle_fwd_dsn});"; $self->{dbhdest}->do($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } # Create the user mapping if it not exists my $usrlbl = 'user'; $usrlbl = 'username' if ($self->{is_mysql} || $self->{is_mssql}); my $sql = "CREATE USER MAPPING IF NOT EXISTS FOR \"$self->{pg_user}\" SERVER $self->{fdw_server} OPTIONS ($usrlbl '$self->{oracle_user}', password '$self->{oracle_pwd}');"; if ($self->{oracle_user} eq "__SEPS__" && $self->{oracle_pwd} eq "__SEPS__") # Replace with empty credentials for an Oracle Wallet connection { $sql =~ s/__SEPS__//g; } $self->{dbhdest}->do($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); } sub _select_foreign_objects { my $self = shift; # With reports we don't have object name limitation return if ($self->{type} ne 'TEST_DATA'); my $str = ''; my @limit_to = (); my @except = (); for (my $j = 0; $j <= $#{$self->{limited}{TABLE}}; $j++) { push(@limit_to, '"' . uc($self->{limited}{TABLE}->[$j]) . '"'); } if ($#limit_to == -1) { for (my $j = 0; $j <= $#{$self->{excluded}{TABLE}}; $j++) { push(@except, '"'. uc($self->{excluded}{TABLE}->[$j] . '"')); } if ($#except > -1) { $str = " EXCEPT ( " . join(', ', @except) . ")"; } } else { $str = " LIMIT TO ( " . join(', ', @limit_to) . ")"; } return $str; } sub _import_foreign_schema { my $self = shift; # Drop and recreate the import schema $self->{dbhdest}->do("DROP SCHEMA $self->{pg_supports_ifexists} $self->{fdw_import_schema} CASCADE") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); $self->{dbhdest}->do("CREATE SCHEMA $self->{fdw_import_schema}") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); # Import foreign table into the dedicated schema $self->{fdw_import_schema} my $sql = "IMPORT FOREIGN SCHEMA \"\U$self->{schema}\E\""; if ($self->{is_mysql}) { $sql = "IMPORT FOREIGN SCHEMA $self->{schema}"; } # ALLOW/EXCLUDE must be applied for data validation $sql .= $self->_select_foreign_objects(); $sql .= " FROM SERVER $self->{fdw_server} INTO $self->{fdw_import_schema}"; if ($self->{is_mssql}) { $sql .= " OPTIONS (import_default 'true')"; } elsif (!$self->{is_mysql}) { $sql .= " OPTIONS (case 'keep', readonly 'true')"; } $self->{dbhdest}->do($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . ", SQL: $sql\n", 0, 1); } sub _data_validation { my $self = shift; # Get all tables information specified by the DBI method table_info $self->logit("Data validation between source database and PostgreSQL...\n", 1); my $unique_clause = ' AND i.indkey IS NOT NULL AND i.indisunique ORDER BY i.indisprimary DESC'; $unique_clause = '' if (!$self->{data_validation_ordering}); # First of all extract all tables from PostgreSQL database with the # unique index column list they must be part of a single schema. my $schema_clause = $self->get_schema_condition(); my $sql = qq{ SELECT c.relname,n.nspname,i.indkey FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace LEFT JOIN pg_catalog.pg_index i ON i.indrelid = c.oid WHERE c.relkind IN ('r','p') AND regexp_match(i.indkey::text, '(^0 | 0 | 0\$|^0\$)') IS NULL $schema_clause $unique_clause }; $self->logit("Get list of table with unique key: $sql\n", 1); my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from catalog about tables."); next; } my %list_tables = (); while ( my @row = $s->fetchrow()) { $row[2] =~ s/ /,/g; $list_tables{"\L$row[0]\E"}{ucols} = $row[2]; $list_tables{"\L$row[0]\E"}{schema} = $row[1]; } $s->finish(); my @foreign_tables = (); if ($self->{fdw_server}) { # Extract all foreign tables imported in schema $self->{fdw_import_schema}. # Normally the table list have already been filtered. $sql = qq{ SELECT c.relname FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = 'f' and n.nspname = '$self->{fdw_import_schema}' }; $self->logit("Get list of foreign tables imported: $sql\n") if ($self->{debug}); $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); if (not $s->execute()) { push(@errors, "Can not extract information from foreign tables."); next; } while ( my @row = $s->fetchrow()) { push(@foreign_tables, $row[0]); } $s->finish(); } else { # Retrieve tables informations foreach my $table (keys %{$self->{tables}}) { push(@foreign_tables, $table); } } # Sort foreign_tables to always have the same ordered iteration @foreign_tables = sort @foreign_tables; my $total_tables = 0; foreach my $f (@foreign_tables) { next if (!exists $list_tables{"\L$f\E"}); $total_tables++; } $self->{child_count} = 0; my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); unlink("${dirprefix}data_validation.log"); print STDERR "Result will be written to file ${dirprefix}data_validation.log\n"; my $tfh = $self->append_export_file($dirprefix . 'data_validation.log', 1); flock($tfh, 2) || die "FATAL: can't lock file ${dirprefix}data_validation.log\n"; $tfh->print("[DATA VALIDATION]\n"); $self->close_export_file($tfh, 1); my $q = 1; foreach my $f (@foreign_tables) { next if (!exists $list_tables{"\L$f\E"}); if ($self->{parallel_tables} > 1) { while ($self->{child_count} >= $self->{parallel_tables}) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } spawn sub { $self->compare_data($f, $list_tables{"\L$f\E"}{ucols}, $list_tables{"\L$f\E"}{schema}); }; $self->{child_count}++; } else { $self->compare_data($f, $list_tables{"\L$f\E"}{ucols}, $list_tables{"\L$f\E"}{schema}); } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($q, $total_tables, 25, '=', 'tables', "checked table: $f" ), "\r"; } $q++; } if ($self->{parallel_tables} > 1) { # Wait for all child end while ($self->{child_count} > 0) { my $kid = waitpid(-1, WNOHANG); if ($kid > 0) { $self->{child_count}--; delete $RUNNING_PIDS{$kid}; } usleep(50000); } } if (!$self->{quiet} && !$self->{debug}) { print STDERR $self->progress_bar($q-1, $total_tables, 25, '=', 'tables', "checked" ), "\n\n"; } } sub compare_data { my ($self, $tb, $ucols, $schema) = @_; my $lbl = 'ORACLEDB'; $lbl = 'MYSQL_DB' if ($self->{is_mysql}); $lbl = 'MSSQL_DB' if ($self->{is_mssql}); my $dbhora = undef; my $dbhpg = undef; if ($self->{fdw_server}) { $self->logit("DEBUG: cloning connection to PostgreSQL.\n", 1); $dbhora = $self->{dbhdest}->clone(); $dbhpg = $self->{dbhdest}->clone(); # Force execution of initial command on both side $self->_ora_initial_command($dbhora); $self->_pg_initial_command($dbhpg); } else { $self->logit("DEBUG: cloning connection to Oracle.\n", 1); $dbhora = $self->{dbh}->clone(); $dbhpg = $self->{dbhdest}->clone(); # Force execution of initial command on both side $self->_ora_initial_command($dbhora); $self->_pg_initial_command($dbhpg); if (!$self->{is_mysql} && !$self->{is_mssql}) { $dbhora->{'LongReadLen'} = $self->{longreadlen}; $dbhora->{'LongTruncOk'} = $self->{longtruncok}; $dbhora->{'ora_objects'} = 0; $self->_datetime_format($dbhora); $self->_numeric_format($dbhora); } else { if ($self->{nls_lang}) { if ($self->{debug} && !$quiet) { $self->logit("Set default encoding to '$self->{nls_lang}' and collate to '$self->{nls_nchar}'\n", 1); } my $collate = ''; $collate = " COLLATE '$self->{nls_nchar}'" if ($self->{nls_nchar}); $sth = $dbhora->prepare("SET NAMES '$self->{nls_lang}'$collate") or $self->logit("FATAL: " . $dbhora->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbhora->errstr . "\n", 0, 1); $sth->finish; } } } my $search_path = $self->set_search_path(); if ($search_path) { $dbhpg->do($search_path) or $self->logit("FATAL: " . $dbhpg->errstr . "\n", 0, 1); } my $sth = undef; if ($self->{fdw_server}) { # Oracle lookup through foreign table $sql = "SELECT a.* FROM $self->{fdw_import_schema}.\"$tb\" a"; if (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}) { $sql .= ' WHERE '; if (!$self->{is_mysql} || ($self->{where}{"\L$table\E"} !~ /\s+LIMIT\s+\d/)) { $sql .= '(' . $self->{where}{"\L$table\E"} . ')'; } else { $sql .= $self->{where}{"\L$table\E"}; } } elsif ($self->{global_where}) { $sql .= ' WHERE '; if (!$self->{is_mysql} || ($self->{global_where} !~ /\s+LIMIT\s+\d/)) { $sql .= '(' . $self->{global_where} . ')'; } else { $sql .= $self->{global_where}; } } $sql .= " ORDER BY " . $ucols if ($self->{data_validation_ordering}); $sql .= " LIMIT $self->{data_validation_rows}" if ($self->{data_validation_rows}); $self->logit("Get rows from foreign tables: $sql\n", 1); $sth = $dbhora->prepare($sql) or $self->logit("ERROR: " . $dbhora->errstr . "\n", 0, 0); $sth->execute or $self->logit("FATAL: " . $dbhora->errstr . "\n", 0, 0); } else { # Extract all column information used to determine data export. # This hash will be used in function _howto_get_data() %{$self->{colinfo}} = $self->_column_attributes($tb, $self->{schema}, 'TABLE'); $sql = $self->_get_oracle_test_data($tb); # Oracle data lookup if ($self->{data_validation_rows}) { if ($self->{db_version} =~ /Release (8|9|10|11)/) { $sql .= " ORDER BY " . $ucols if ($self->{data_validation_ordering}); $sql = "SELECT * FROM ( $sql ) WHERE ROWNUM <= $self->{data_validation_rows}"; } else { $sql .= " ORDER BY " . $ucols if ($self->{data_validation_ordering}); if (!$self->{is_mysql}) { $sql .= " FETCH FIRST $self->{data_validation_rows} ROWS ONLY"; } else { $sql .= " LIMIT $self->{data_validation_rows}"; } } } elsif ($self->{data_validation_ordering}) { $sql .= " ORDER BY " . $ucols; } $self->logit("Get rows from oracle tables: $sql\n", 1); $sth = $dbhora->prepare($sql) or $self->logit("ERROR: " . $dbhora->errstr . "\n", 0, 0); $sth->execute or $self->logit("FATAL: " . $dbhora->errstr . "\n", 0, 0); } #### # PostgreSQL lookup #### # Compute the target list to be able to call functions to format some data type like geometry my $tname = $self->quote_object_name($schema) . '.'. $self->quote_object_name($tb); my $tmp = qq{ SELECT attname,attnum,atttypid::regtype FROM pg_attribute WHERE attrelid = '$tname'::regclass AND attnum > 0 AND NOT attisdropped ORDER BY attnum}; my $tmpsth = $dbhpg->prepare($tmp); $tmpsth->execute(); my @tlist = (); my @dest_types = (); while ( my @crow = $tmpsth->fetchrow()) { next if (!$self->is_in_struct($tb, $crow[0])); if ($crow[2] eq 'geometry') { if ($self->{is_mysql}) { push(@tlist, "$self->{st_astext_function}($crow[0])"); } else { push(@tlist, $crow[0]); } } else { push(@tlist, $crow[0]); } push(@dest_types, $crow[2]); } $tmpsth->finish(); # Quote column name when required for (my $i = 0; $i <= $#tlist; $i++) { if ($tlist[$i] !~ /"/ && $self->is_reserved_words($tlist[$i])) { $tlist[$i] = '"' . $tlist[$i] . '"'; } if ($self->{preserve_case}) { $tlist[$i] =~ s/^(["]*)/"/; $tlist[$i] =~ s/(["]*)$/"/; } } # Now get the data my $sql2 = "SELECT " . join(',', @tlist) . " FROM " . $self->quote_object_name($schema) . '.'. $self->quote_object_name($tb); $sql2 .= " ORDER BY " . $ucols if ($self->{data_validation_ordering}); $sql2 .= " LIMIT $self->{data_validation_rows}" if ($self->{data_validation_rows}); $self->logit("Get rows from migrated tables: $sql2\n") if ($self->{debug}); my $sth2 = $dbhpg->prepare($sql2) or $self->logit("ERROR: " . $dbhpg->errstr . "\n", 0, 0); $sth2->execute or $self->logit("FATAL: " . $dbhpg->errstr . "\n", 0, 0); my $i = 1; $self->logit("Checking data validation for table $f\n", 1); my $nerror = 0; my $error_msg = ''; while ( my @orow = $sth->fetchrow()) { my @prow = $sth2->fetchrow(); # There is an issue to compare timestamp, try to fix it for (my $i = 0; $i <= $#orow; $i++) { # We must adjust the microsecond information on PG data if ( $self->{enable_microsecond} && exists $self->{colinfo}{$tb} && $self->{colinfo}{$tb}{data_type}{$i+1} =~ /^(DATE|TIMESTAMP)/i ) { if ($orow[$i] =~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+$/ && $prow[$i] =~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/) { $prow[$i] .= '.000000'; } elsif ($prow[$i] =~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+$/) { while (length($prow[$i]) < 26) { $prow[$i] .= '0' }; } } # RAW(16) and RAW(32) might have been replaced by uuid if ($dest_types[$i] eq 'uuid' && $self->{colinfo}{$tb}{data_type}{$i+1} eq 'RAW') { $orow[$i] =~ s/^([A-F0-9]{8})([A-F0-9]{4})([A-F0-9]{4})([A-F0-9]{4})([A-F0-9]{12})$/\L$1-$2-$3-$4-$5\E/i; } # Cover boolean transformation if ($dest_types[$i] eq 'boolean') { foreach my $k (keys %{ $self->{ora_boolean_values} }) { if ($orow[$i] =~ /^$k$/i) { $orow[$i] = $self->{ora_boolean_values}{$k}; last; } } if ($prow[$i] == 1) { $prow[$i] = 't'; } elsif (defined $prow[$i]) { $prow[$i] = 'f'; } } # Remove extra zero following the decimal when needed if ($dest_types[$i] =~ /(double precision|real|numeric)/i) { $orow[$i] =~ s/\.[0]+$// if ($prow[$i] !~ /\.[0]+$/); if ($prow[$i] !~ /\.[0-9]+[0]+$/) { while ($orow[$i] =~ s/(\.[0-9]+)[0]+$/$1/) {}; } $prow[$i] =~ s/\.[0]+$// if ($orow[$i] !~ /\.[0]+$/); if ($orow[$i] !~ /\.[0-9]+[0]+$/) { while ($prow[$i] =~ s/(\.[0-9]+)[0]+$/$1/) {}; } } # MySQL remove the trailing space at end of char(n) -> take care of that in your app # PostgreSQL keep the trailing spaces in respect to SQL standard. if ($self->{is_mysql} && $self->{colinfo}{$tb}{data_type}{$i+1} =~ /^CHAR/i) { $prow[$i] =~ s/[ ]+$//; } # Oracle can report decimal as .nn, PG always have a 0 at startup if ($self->{colinfo}{$tb}{data_type}{$i+1} eq 'NUMBER') { $orow[$i] =~ s/^([\-]?)(\.\d+)/${1}0$2/; } } my $ora_data = join('|', @orow); my $pg_data = join('|', @prow); if ($ora_data ne $pg_data) { $error_msg .= "$lbl:$tb:$i:[$ora_data]\n"; $error_msg .= "POSTGRES:$tb:$i:[$pg_data]\n"; $nerror++; } $i++; last if ($nerror == $self->{data_validation_error}); } $sth->finish; $sth2->finish; $dbhora->disconnect() if ($dbhora); $dbhpg->disconnect() if ($dbhpg); my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $tfh = $self->append_export_file($dirprefix . 'data_validation.log', 1); flock($tfh, 2) || die "FATAL: can't lock file data_validation.log\n"; $tfh->print("Data validation for table $tb: " . ((!$nerror) ? "OK\n" : "$nerror FAIL\n")); if ($error_msg) { $tfh->print("-----------------------------------------------------------------\n"); $tfh->print($error_msg); $tfh->print("-----------------------------------------------------------------\n"); } $self->close_export_file($tfh, 1); } #### # Set query to extract Oracle table content for data comparison #### sub _get_oracle_test_data { my ($self, $table, $part_name, $is_subpart) = @_; # Rename table and double-quote it if required my $tmptb = ''; # Prefix partition name with tablename, if pg_supports_partition is enabled # direct import to partition is not allowed so import to main table. if (!$self->{pg_supports_partition} && $part_name && $self->{rename_partition}) { $tmptb = $self->get_replaced_tbname($table . '_' . $part_name); } elsif (!$self->{pg_supports_partition} && $part_name) { $tmptb = $self->get_replaced_tbname($part_name || $table); } else { $tmptb = $self->get_replaced_tbname($table); } # Build the header of the query my @tt = (); my @stt = (); my @nn = (); my $col_list = ''; # Extract column information following the Oracle position order my @fname = (); my (@pg_colnames_nullable, @pg_colnames_notnull, @pg_colnames_pkey); foreach my $i ( 0 .. $#{$self->{tables}{$table}{field_name}} ) { my $fieldname = ${$self->{tables}{$table}{field_name}}[$i]; next if (!$self->is_in_struct($table, $fieldname)); my $f = $self->{tables}{"$table"}{column_info}{"$fieldname"}; $f->[2] =~ s/\D//g; if (!$self->{enable_blob_export} && $f->[1] =~ /blob/i) { # user don't want to export blob next; } if (!$self->{enable_clob_export} && $f->[1] =~ /clob/i) { # user don't want to export clob next; } my $is_pk = $self->is_primary_key_column($table, $fieldname); # When lo_import is used we only want the PK colmuns and the BLOB if ($self->{lo_import} && $f->[1] !~ /blob/i && !$is_pk) { next; } if (!$self->{preserve_case}) { push(@fname, lc($fieldname)); } else { push(@fname, $fieldname); } my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4]); $type = "$f->[1], $f->[2]" if (!$type); if (uc($f->[1]) eq 'ENUM') { my $keyname = lc($table . '_' . $colname . '_t'); $f->[1] = $keyname; } push(@stt, uc($f->[1])); push(@tt, $type); push(@nn, $f->[0]); # Change column names my $colname = $f->[0]; if ($self->{replaced_cols}{lc($table)}{lc($f->[0])}) { $self->logit("\tReplacing column $f->[0] as " . $self->{replaced_cols}{lc($table)}{lc($f->[0])} . "...\n", 1); $colname = $self->{replaced_cols}{lc($table)}{lc($f->[0])}; } $colname = $self->quote_object_name($colname); if ($colname !~ /"/ && $self->is_reserved_words($colname)) { $colname = '"' . $colname . '"'; } $col_list .= "$colname,"; if ($is_pk) { push @pg_colnames_pkey, "$colname"; } elsif ($f->[3] =~ m/^Y/) { push @pg_colnames_nullable, "$colname"; } else { push @pg_colnames_notnull, "$colname"; } } $col_list =~ s/,$//; $self->{tables}{$table}{pg_colnames_nullable} = \@pg_colnames_nullable; $self->{tables}{$table}{pg_colnames_notnull} = \@pg_colnames_notnull; $self->{tables}{$table}{pg_colnames_pkey} = \@pg_colnames_pkey; # Extract all data from the current table my $query = $self->_howto_get_data($table, \@nn, \@tt, \@stt, $part_name, $is_subpart); return $query } 1; __END__ =head1 AUTHOR Gilles Darold =head1 COPYRIGHT Copyright (c) 2000-2025 Gilles Darold - All rights reserved. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see < http://www.gnu.org/licenses/ >. =head1 SEE ALSO L, L =cut ora2pg-25.0/lib/Ora2Pg/000077500000000000000000000000001500113072400144635ustar00rootroot00000000000000ora2pg-25.0/lib/Ora2Pg/GEOM.pm000066400000000000000000000672031500113072400155600ustar00rootroot00000000000000package Ora2Pg::GEOM; #------------------------------------------------------------------------------ # Project : Oracle to PostgreSQL database schema converter # Name : Ora2Pg/GEOM.pm # Language : Perl # Authors : Gilles Darold, gilles _AT_ darold _DOT_ net # Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved - # Function : Perl module used to convert Oracle SDO_GEOMETRY into PostGis # Usage : See documentation #------------------------------------------------------------------------------ # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see < http://www.gnu.org/licenses/ >. # #------------------------------------------------------------------------------ # # Most of this work is inspired from the JTS Topology Suite developed by # Vivid Solutions, Inc. (http://www.vividsolutions.com/jts/JTSHome.htm) # JTS is an open source (under the LGPL license) Java library. # See http://www.gnu.org/copyleft/lesser.html for the license. # #------------------------------------------------------------------------------ # # Special thanks to Dominique Legendre and The French Geological Survey - BRGM # http://www.brgm.eu/ and Olivier Picavet from Oslandia http://www.oslandia.com/ # who help me a lot with spatial understanding and their testing efforts. # #------------------------------------------------------------------------------ use vars qw($VERSION); use strict; $VERSION = '25.0'; # SDO_ETYPE # Second element of triplet in SDO_ELEM_INFO my %SDO_ETYPE = ( # code representing Point 'POINT' => 1, # code representing Line 'LINESTRING' => 2, # code representing Polygon 'POLYGON' => 3, # code representing compound line 'COMPOUNDCURVE' => 4, # code representing exterior counterclockwise polygon ring 'POLYGON_EXTERIOR' => 1003, # code representing interior clockwise polygon ring 'POLYGON_INTERIOR' => 2003, # code repersenting compound polygon counterclockwise polygon ring 'COMPOUND_POLYGON_EXTERIOR' => 1005, # code repersenting compound polygon clockwise polygon ring 'COMPOUND_POLYGON_INTERIOR' => 2005, ); # SDO_GTYPE # Type of the geometry my %SDO_GTYPE = ( # Point 'POINT' => 1, # Line or Curve 'LINESTRING' => 2, # Polygon 'POLYGON' => 3, # Geometry collection 'GEOMETRYCOLLECTION' => 4, # Multpoint 'MULTIPOINT' => 5, # Multiline or Multicurve 'MULTILINESTRING' => 6, # Multipolygon 'MULTIPOLYGON' => 7 ); # SDO_INTERPRETATIONS # Third element of triplet in SDO_ELEM_INFO # applies to points - sdo_etype 1 my %INTERPRETATION_POINT = ( '0' => 'ORIENTED_POINT', '1' => 'SIMPLE_POINT' # n > 1: point cluster with n points ); # applies to lines - sdo_etype 2 my %INTERPRETATION_LINE = ( '1' => 'STRAIGHT_SEGMENTS', '2' => 'CURVED_SEGMENTS' ); # applies to polygons - sdo_etypes 1003 and 2003 my %INTERPRETATION_MULTI = ( '1' => 'SIMPLE_POLY', '2' => 'ARCS_POLY', '3' => 'RECTANGLE', '4' => 'CIRCLE' ); sub new { my ($class, %options) = @_; # This create an OO perl object my $self = {}; bless ($self, $class); # Initialize this object $self->_init(%options); # Return the instance return($self); } sub _init { my ($self, %opt) = @_; $self->{dimension} = $opt{dimension} || -1; $self->{srid} = $opt{srid} || -1; $self->{geometry} = undef; } sub parse_sdo_geometry { my ($self, $sdo_geom) = @_; # SDO_GEOMETRY DEFINITION # CREATE TYPE sdo_geometry AS OBJECT ( # SDO_GTYPE NUMBER, # SDO_SRID NUMBER, # SDO_POINT SDO_POINT_TYPE, # SDO_ELEM_INFO SDO_ELEM_INFO_ARRAY, # SDO_ORDINATES SDO_ORDINATE_ARRAY # ); #CREATE TYPE sdo_point_type AS OBJECT ( # X NUMBER, # Y NUMBER, # Z NUMBER); #CREATE TYPE sdo_elem_info_array AS VARRAY (1048576) of NUMBER; #CREATE TYPE sdo_ordinate_array AS VARRAY (1048576) of NUMBER; # # SDO_ELEM_INFO # Each triplet set of numbers is interpreted as follows: # SDO_STARTING_OFFSET -- Indicates the offset within the SDO_ORDINATES array where the first # ordinate for this element is stored. Offset values start at 1 and not at 0. # SDO_ETYPE -- Indicates the type of the element. # SDO_interpretation -- Means one of two things, depending on whether or not SDO_ETYPE is a compound element. # If SDO_ETYPE is a compound element (4, 1005, or 2005), this field specifies how many subsequent # triplet values are part of the element. # If the SDO_ETYPE is not a compound element (1, 2, 1003, or 2003), the interpretation attribute determines how # the sequence of ordinates for this element is interpreted. return undef if ($#{$sdo_geom} < 0); # Get dimension and geometry type if ($sdo_geom->[0] =~ /^(\d)(\d)(\d{2})$/) { $self->{geometry}{sdo_gtype} = $sdo_geom->[0] || 0; # Extract the geometry dimension this is represented as the leftmost digit $self->{geometry}{dim} = $1; # Extract the linear referencing system $self->{geometry}{lrs} = $2; # Extract the geometry template type this is represented as the rightmost two digits $self->{geometry}{gtype} = $3; if ($self->{geometry}{dim} < 2) { $self->logit("ERROR: Dimension $self->{geometry}{dim} is not valid. Either specify a dimension or use Oracle Locator Version 9i or later.\n"); return undef; } } else { $self->logit("ERROR: wrong SDO_GTYPE format in SDO_GEOMETRY data\n", 0, 0); return undef; } # Set EWKT geometry dimension $self->{geometry}{suffix} = ''; if ($self->{geometry}{dim} == 3) { $self->{geometry}{suffix} = 'Z'; } elsif ($self->{geometry}{dim} == 4) { $self->{geometry}{suffix} = 'ZM'; } # Get the srid from the data otherwise it will be # overriden by the column srid found in meta information $self->{geometry}{srid} = $sdo_geom->[1] if (defined $sdo_geom->[1] && $sdo_geom->[1] ne ''); $self->{geometry}{srid} = $self->{srid} if ($self->{geometry}{srid} eq ''); # Look at point only coordonate @{$self->{geometry}{sdo_point}} = (); if ($sdo_geom->[2] =~ /^ARRAY\(0x/) { map { if (/^[-\d]+$/) { s/,/\./; s/$/\.0/; } } @{$sdo_geom->[2]}; push(@{$self->{geometry}{sdo_point}}, @{$sdo_geom->[2]}); } # Extract elements info by triplet @{$self->{geometry}{sdo_elem_info}} = (); if ($sdo_geom->[3] =~ /^ARRAY\(0x/) { push(@{$self->{geometry}{sdo_elem_info}}, @{$sdo_geom->[3]}); } # Extract ordinates information as arrays of dimension elements @{$self->{geometry}{sdo_ordinates}} = (); if ($sdo_geom->[4] =~ /^ARRAY\(0x/) { map { if (/^[-\d]+$/) { s/,/\./; s/$/\.0/; } } @{$sdo_geom->[4]}; push(@{$self->{geometry}{sdo_ordinates}}, @{$sdo_geom->[4]}); } return $self->extract_geometry(); } # Extract geometries sub extract_geometry { my ($self, %geometry) = @_; my @coords = (); # Extract coordinates following the dimension if ( ($self->{geometry}{gtype} == 1) && ($#{$self->{geometry}{sdo_point}} >= 0) && ($#{$self->{geometry}{sdo_elem_info}} == -1) ) { # Single Point Type Optimization @coords = $self->coordinates(@{$self->{geometry}{sdo_point}}); @{$self->{geometry}{sdo_elem_info}} = ( 1, $SDO_ETYPE{POINT}, 1 ); } else { @coords = $self->coordinates(@{$self->{geometry}{sdo_ordinates}}); } # Get the geometry if ($self->{geometry}{gtype} == $SDO_GTYPE{POINT}) { return $self->createPoint(0, \@coords); } if ($self->{geometry}{gtype} == $SDO_GTYPE{LINESTRING}) { if ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUNDCURVE}) { return $self->createCompoundLine(1, \@coords, -1); } else { return $self->createLine(0, \@coords); } } if ($self->{geometry}{gtype} == $SDO_GTYPE{POLYGON}) { return $self->createPolygon(0, \@coords); } if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTIPOINT}) { return $self->createMultiPoint(0, \@coords); } if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTILINESTRING}) { return $self->createMultiLine(0, \@coords, -1); } if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTIPOLYGON}) { return $self->createMultiPolygon(0, \@coords, -1); } if ($self->{geometry}{gtype} == $SDO_GTYPE{GEOMETRYCOLLECTION}) { return $self->createCollection(0, \@coords,-1); } } # Build an array of references arrays of coordinates following the dimension sub coordinates { my ($self, @ordinates) = @_; my @coords = (); my @tmp = (); # The number of ordinates per coordinate is taken from the dimension for (my $i = 1; $i <= $#ordinates + 1; $i++) { push(@tmp, $ordinates[$i - 1]); if ($i % $self->{geometry}{dim} == 0) { push(@coords, [(@tmp)]); @tmp = (); } } return @coords; } # Accesses the starting index in the ordinate array for the current geometry sub get_start_offset { my ($self, $t_idx) = @_; if ((($t_idx * 3) + 0) >= ($#{$self->{geometry}{sdo_elem_info}} + 1)) { return -1; } return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 0]; } # Get the SDO_ETYPE part from the elemInfo triplet sub eType { my ($self, $t_idx) = @_; if ((($t_idx * 3) + 1) >= ($#{$self->{geometry}{sdo_elem_info}}+1)) { return -1; } return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 1]; } # Get the interpretation part the elemInfo triplet sub interpretation { my ($self, $t_idx) = @_; if ((($t_idx * 3) + 2) >= ($#{$self->{geometry}{sdo_elem_info}}+1)) { return -1; } return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 2]; } # Create Geometry Collection as encoded by elemInfo. sub createCollection { my ($self, $elemIndex, $coords, $numGeom) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $length = ($#{$coords}+1) * $self->{geometry}{dim}; if ($sOffset > $length) { $self->logit("ERROR: SDO_ELEM_INFO for Collection starting offset $sOffset inconsistent with ordinates length $length"); } my $endTriplet = ($#{$self->{geometry}{sdo_elem_info}}+1) / 3; my @list_geom = (); my $etype; my $interpretation; my $geom; my $cont = 1; for (my $i = $elemIndex; $cont && $i < $endTriplet; $i++) { $etype = $self->eType($i); $interpretation = $self->interpretation($i); # Exclude type 0 (zero) element if ($etype == 0) { $self->logit("WARNING: SDO_ETYPE $etype not supported EWKT Geometry by Ora2Pg. Check what's going wrong with this geometry."); next; } if ($etype == -1) { $cont = 0; # We reach the end of the list - get out of here next; } elsif ($etype == $SDO_ETYPE{POINT}) { if ($interpretation == 1) { $geom = $self->createPoint($i, $coords); } elsif ($interpretation > 1) { $geom = $self->createMultiPoint($i, $coords); } } elsif ($etype == $SDO_ETYPE{LINESTRING}) { $geom = $self->createLine($i, $coords); } elsif ( ($etype == $SDO_ETYPE{POLYGON}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) || ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) ) { $geom = $self->createPolygon($i, $coords); } elsif ( ($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) || ($etype == $SDO_ETYPE{COMPOUND_POLYGON_INTERIOR}) ) { $geom = $self->createCompoundPolygon($i, $coords); # Skip elements $i += $interpretation; } else { $self->logit("ERROR: SDO_ETYPE $etype not representable as a EWKT Geometry by Ora2Pg."); next; } push(@list_geom, $geom); } return "GEOMETRYCOLLECTION$self->{geometry}{suffix} (" . join(', ', @list_geom) . ')'; } # Create MultiPolygon sub createMultiPolygon { my ($self, $elemIndex, $coords, $numGeom) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; if (($sOffset < 1) || ($sOffset > $length)) { $self->logit("ERROR: SDO_ELEM_INFO for MultiPolygon starting offset $sOffset inconsistent with ordinates length $length"); } # For SDO_ETYPE values 1003 and 2003, the first digit indicates exterior (1) or interior (2) if (($etype != $SDO_ETYPE{POLYGON}) && ($etype != $SDO_ETYPE{POLYGON_EXTERIOR})) { $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POLYGON or POLYGON_EXTERIOR"); } if (($interpretation != 1) && ($interpretation != 3)) { return undef; } my $endTriplet = ($numGeom != -1) ? $elemIndex + $numGeom : (($#{$self->{geometry}{sdo_elem_info}}+1) / 3) + 1; my @list = (); my $cont = 1; for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1; $i++) { # Exclude type 0 (zero) element next if ($etype == 0); if (($etype == $SDO_ETYPE{POLYGON}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR})) { my $poly = $self->createPolygon($i, $coords); $poly =~ s/POLYGON$self->{geometry}{suffix} //; if ($etype != $self->eType($i-1)) { if ( ($etype = $SDO_ETYPE{POLYGON_INTERIOR}) && ($SDO_ETYPE{POLYGON_EXTERIOR} == $self->eType($i-1)) ) { $poly =~ s/^\(//; $list[-1] =~ s/\)$//; } } push(@list, $poly); # Skip interior rings while ($self->eType($i+1) == $SDO_ETYPE{POLYGON_INTERIOR}) { $i++; } } else { # not a Polygon - get out here $cont = 0; } } return "MULTIPOLYGON$self->{geometry}{suffix} (" . join(', ', @list) . ')'; } # Create MultiLineString sub createMultiLine { my ($self, $elemIndex, $coords, $numGeom) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; if (($sOffset < 1) || ($sOffset > $length)) { $self->logit("ERROR: SDO_ELEM_INFO for MultiLine starting offset $sOffset inconsistent with ordinates length $length"); } if ($etype != $SDO_ETYPE{LINESTRING}) { $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected LINESTRING"); } my $endTriplet = ($numGeom != -1) ? ($elemIndex + $numGeom) : (($#{$self->{geometry}{sdo_elem_info}} + 1) / 3); my @list = (); my $cont = 1; for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1 ; $i++) { # Exclude type 0 (zero) element next if ($etype == 0); if ($etype == $SDO_ETYPE{LINESTRING}) { push(@list, $self->createLine($i, $coords)); } elsif ($etype == $SDO_ETYPE{COMPOUNDCURVE}) { push(@list, $self->createCompoundLine(1, $coords, -1)); } else { # not a LineString - get out of here $cont = 0; } } if ($interpretation > 1 || grep(/CIRCULARSTRING/, @list)) { return "MULTICURVE$self->{geometry}{suffix} (" . join(', ', @list) . ')'; } map { s/LINESTRING$self->{geometry}{suffix} //; } @list; return "MULTILINESTRING$self->{geometry}{suffix} (" . join(', ', @list) . ')'; } # Create MultiPoint sub createMultiPoint { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; if (($sOffset < 1) || ($sOffset > $length)) { $self->logit("ERROR: SDO_ELEM_INFO for MultiPoint starting offset $sOffset inconsistent with ordinates length $length"); } if ($etype != $SDO_ETYPE{POINT}) { $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POINT"); } my @point = (); my $start = ($sOffset - 1) / $self->{geometry}{dim}; if ($interpretation > 1) { for (my $i = $start + 1; $i <= $interpretation; $i++) { push(@point, $self->setCoordicates($coords, $i, $i)); } # Oriented point are not supported by WKT } elsif ($interpretation != 0) { # There is multiple single point my $cont = 1; for (my $i = $start + 1; $cont && ($etype = $self->eType($i - 1)) != -1; $i++) { # Exclude type 0 (zero) element next if ($etype == 0); next if ($self->interpretation($i - 1) == 0); if ($etype == $SDO_ETYPE{POINT}) { push(@point, $self->setCoordicates($coords, $i, $i)); } else { $cont = 0; } } } my $points = "MULTIPOINT$self->{geometry}{suffix} ((" . join('), (', @point) . '))'; return $points; } # Create Polygon sub createPolygon { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } if ( ($sOffset < 1) || ($sOffset > ($#{$coords} + 1) * $self->{geometry}{dim}) ) { $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with COORDINATES length " . (($#{$coords} + 1) * $self->{geometry}{dim}) ); } my $poly = ''; if ($interpretation == 1 ) { $poly = "POLYGON$self->{geometry}{suffix} (" . $self->createLinearRing($elemIndex, $coords).")"; } elsif ($interpretation == 2) { $poly = "CURVEPOLYGON$self->{geometry}{suffix} (" . $self->createLinearRing($elemIndex, $coords).")"; } elsif ($interpretation == 3) { $poly = "POLYGON$self->{geometry}{suffix} (" . $self->createRectangle($elemIndex, $coords).")"; } else { $self->logit("ERROR: Unsupported polygon type with interpretation $interpretation probably mangled"); $poly = "POLYGON$self->{geometry}{suffix} (" . $self->createLinearRing($elemIndex, $coords).")"; } return $poly; } # Create CompoundPolygon # AD: Unsure whether my dataset has testcases for this sub createCompoundPolygon { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); if ( ($sOffset < 1) || ($sOffset > ($#{$coords} + 1) * $self->{geometry}{dim}) ) { $self->logit("ERROR: SDO_ELEM_INFO for Compound Polygon starting offset $sOffset inconsistent with COORDINATES length " . (($#{$coords} + 1) * $self->{geometry}{dim}) ); } my @rings = (); my $cont = 1; for (my $i = $elemIndex+1; $cont && ($etype = $self->eType($i)) != -1; $i++) { # Exclude type 0 (zero) element next if ($etype == 0); if ($etype == $SDO_ETYPE{LINESTRING}) { push(@rings, $self->createLinearRing($i, $coords)); } else { $self->logit("ERROR: ETYPE $etype inconsistent with Compound Polygon" ); if ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) { push(@rings, $self->createLinearRing($i, $coords)); } elsif ($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) { next; } elsif ($etype == $SDO_ETYPE{POLYGON}) { push(@rings, $self->createLinearRing($i, $coords)); } else { # not a LinearRing - get out of here $cont = 0; } } } return "POLYGON$self->{geometry}{suffix} (" . join(', ', @rings) . ')'; } sub createRectangle { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; if ($sOffset > $length) { $self->logit("ERROR: SDO_ELEM_INFO for Rectangle starting offset $sOffset inconsistent with ordinates length $length"); } my $ring = ''; my $start = ($sOffset - 1) / $self->{geometry}{dim}; my $eOffset = $self->get_start_offset($elemIndex+1); # -1 for end my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); if ($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) { $ring = join(' ', @{$coords->[$start]}). ','.${$coords->[$start+1]}[0].' '.${$coords->[$start]}[1]. ','.join(' ', @{$coords->[$start+1]}). ','.${$coords->[$start]}[0].' '.${$coords->[$start+1]}[1]. ','.join(' ', @{$coords->[$start]}); } else { # INTERIOR $ring = join(' ', @{$coords->[$start]}). ','.${$coords->[$start]}[0].' '.${$coords->[$start+1]}[1]. ','.join(' ', @{$coords->[$start+1]}). ','.${$coords->[$start+1]}[0].' '.${$coords->[$start]}[1]. ','.join(' ', @{$coords->[$start]}); } return '('. $ring. ')'; } # Create Linear Ring for polygon sub createLinearRing { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } # Exclude type 0 (zero) element return if ($etype == 0); if ($sOffset > $length) { $self->logit("ERROR: SDO_ELEM_INFO for LinearRing starting offset $sOffset inconsistent with ordinates length $length"); } if ( ($etype == $SDO_ETYPE{COMPOUND_POLYGON_INTERIOR}) || ($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) ) { return undef; } my $ring = ''; my $start = ($sOffset - 1) / $self->{geometry}{dim}; if (($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) && ($interpretation == 3)) { my $min = $coords->[$start]; my $max = $coords->[$start+1]; $ring = join(' ', @$min) . ', ' . $max->[0] . ' ' . $min->[1] . ', '; $ring .= join(' ', @$max) . ', ' . $min->[0] . ' ' . $max->[1] . ', '; $ring .= join(' ', @$min); } elsif (($etype == $SDO_ETYPE{POLYGON_INTERIOR}) && ($interpretation == 3)) { my $min = $coords->[$start]; my $max = $coords->[$start+1]; $ring = join(' ', @$min) . ', ' . $min->[0] . ' ' . $max->[1] . ', '; $ring .= join(' ', @$max) . ', ' . $max->[0] . ' ' . $min->[1] . ', '; $ring .= join(' ', @$min); } else { my $eOffset = $self->get_start_offset($elemIndex+1); # -1 for end my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); # Polygon have the last point specified exactly the same point as the first, for others # the coordinates for a point designating the end of one arc and the start of the next # arc are not repeated in SDO_GEOMETRY but must be repeated in WKT. if ( ($etype != $SDO_ETYPE{POLYGON}) || ($interpretation != 1) ) { #$end++; } if ($interpretation == 2) { if ( ($etype == $SDO_ETYPE{LINESTRING}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) || ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) ) { #$end++; } } if ( ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUND_POLYGON_INTERIOR}) || ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) ) { $end++; } $ring = $self->setCoordicates($coords, $start+1, $end); if ($interpretation == 4) { # With circle we have to repeat the first coordinates $ring .= ', ' . $self->setCoordicates($coords, $start+1, $start+1); } } if (($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) && ($interpretation == 2)) { $ring = "CIRCULARSTRING$self->{geometry}{suffix} (" . $ring . ')'; } elsif (($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) && ($interpretation == 2)) { $ring = "COMPOUNDCURVE$self->{geometry}{suffix} (" . $ring . ')'; } elsif ( $etype == $SDO_ETYPE{LINESTRING} && ($interpretation == 2)) { $ring = "CIRCULARSTRING$self->{geometry}{suffix} (" . $ring . ')'; } else { $ring = '(' . $ring . ')'; } return $ring; } # Create CompoundLineString sub createCompoundLine { my ($self, $elemIndex, $coords, $numGeom) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); while ($etype == 0) { $elemIndex++; $sOffset = $self->get_start_offset($elemIndex); $etype = $self->eType($elemIndex); $interpretation = $self->interpretation($elemIndex); } my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; if (($sOffset < 1) || ($sOffset > $length)) { $self->logit("ERROR: SDO_ELEM_INFO for CompoundLine starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); } if ($etype != $SDO_ETYPE{LINESTRING}) { $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected LINESTRING"); } my $endTriplet = ($numGeom != -1) ? ($elemIndex + $numGeom) : (($#{$self->{geometry}{sdo_elem_info}} + 1) / 3); my @list = (); my $cont = 1; for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1 ; $i++) { # Exclude type 0 (zero) element next if ($etype == 0); if ($etype == $SDO_ETYPE{LINESTRING}) { push(@list, $self->createLine($i, $coords)); } else { # not a LineString - get out of here $cont = 0; } } return "COMPOUNDCURVE$self->{geometry}{suffix} (" . join(', ', @list) . ')'; } # Create LineString sub createLine { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); if ($etype != $SDO_ETYPE{LINESTRING}) { return undef; } my $start = ($sOffset - 1) / $self->{geometry}{dim}; my $eOffset = $self->get_start_offset($elemIndex + 1); # -1 for end my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); if ( $self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUNDCURVE}) { $end++; } if ($interpretation != 1) { my $line = "CIRCULARSTRING$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; return $line; } my $line = "LINESTRING$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; return $line; } # Create Point sub createPoint { my ($self, $elemIndex, $coords) = @_; my $sOffset = $self->get_start_offset($elemIndex); my $etype = $self->eType($elemIndex); my $interpretation = $self->interpretation($elemIndex); my $length = ($#{$coords}+1) * $self->{geometry}{dim}; if (($sOffset < 1) || ($sOffset > $length)) { $self->logit("ERROR: SDO_ELEM_INFO for Point starting offset $sOffset inconsistent with ordinates length $length"); } if ($etype != $SDO_ETYPE{POINT}) { $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POINT"); } # Point cluster if ($interpretation > 1) { return $self->createMultiPoint($elemIndex, $coords); # Oriented point should be processed by MULTIPOINT } elsif ($interpretation == 0) { $self->logit("ERROR: SDO_ETYPE.POINT with interpretation = 0 is not supported"); return undef; } my $start = ($sOffset - 1) / $self->{geometry}{dim}; my $eOffset = $self->get_start_offset($elemIndex + 1); # -1 for end my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); my $point = "POINT$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; return $point; } sub setCoordicates { my ($self, $coords, $start, $end) = @_; my $str = ''; $start ||= 1; $end = $#{$coords} + 1 if ($end <= 0); for (my $i = $start - 1; $i < $end && ($i <= $#{$coords}); $i++) { my $coordinates = join(' ', @{$coords->[$i]}); if ($coordinates =~ /\d/) { $str .= "$coordinates, "; } } $str =~ s/, $//; return $str; } sub logit { my ($self, $message, $level, $critical) = @_; if (defined $self->{fhlog}) { $self->{fhlog}->print("$message\n"); } else { print "$message\n"; } } 1; ora2pg-25.0/lib/Ora2Pg/MSSQL.pm000066400000000000000000002703471500113072400157350ustar00rootroot00000000000000package Ora2Pg::MSSQL; use vars qw($VERSION); use strict; use DBI; use POSIX qw(locale_h); use Benchmark; #set locale to LC_NUMERIC C setlocale(LC_NUMERIC,"C"); $VERSION = '23.1'; # Some function might be excluded from export and assessment. our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); # These definitions can be overriden from configuration # file using the DATA_TYPË configuration directive. our %SQL_TYPE = ( 'TINYINT' => 'smallint', # 1 byte 'SMALLINT' => 'smallint', # 2 bytes 'INT' => 'integer', # 4 bytes 'BIGINT' => 'bigint', # 8 bytes 'DECIMAL' => 'numeric', 'DEC' => 'numeric', 'NUMERIC' => 'numeric', 'BIT' => 'boolean', 'MONEY' => 'numeric(15,4)', 'SMALLMONEY' => 'numeric(6,4)', 'FLOAT' => 'double precision', 'REAL' => 'real', 'DATE' => 'date', 'SMALLDATETIME' => 'timestamp(0) without time zone', 'DATETIME' => 'timestamp(3) without time zone', 'DATETIME2' => 'timestamp without time zone', 'DATETIMEOFFSET' => 'timestamp with time zone', 'TIME' => 'time without time zone', 'CHAR' => 'char', 'VARCHAR' => 'varchar', 'TEXT' => 'text', 'NCHAR' => 'char', 'NVARCHAR' => 'varchar', 'NTEXT' => 'text', 'VARBINARY' => 'bytea', 'BINARY' => 'bytea', 'IMAGE' => 'bytea', 'UNIQUEIDENTIFIER' => 'uuid', 'ROWVERSION' => 'bytea', 'TIMESTAMP' => 'bytea', # synonym of ROWVERSION 'XML' => 'xml', 'HIERARCHYID' => 'varchar', # The application need to handle the value, no PG equivalent 'GEOMETRY' => 'geometry', 'GEOGRAPHY' => 'geometry', 'SYSNAME' => 'varchar(256)', 'SQL_VARIANT' => 'text' ); sub _db_connection { my $self = shift; $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$self->{quiet}); if (!defined $self->{oracle_pwd}) { eval("use Term::ReadKey;"); if (!$@) { $self->{oracle_user} = $self->_ask_username('MSSQL') unless (defined $self->{oracle_user}); $self->{oracle_pwd} = $self->_ask_password('MSSQL'); } } my $dbh = DBI->connect("$self->{oracle_dsn}", $self->{oracle_user}, $self->{oracle_pwd}, { 'RaiseError' => 1, AutoInactiveDestroy => 1, #odbc_cursortype => 2, PrintError => 0, odbc_utf8_on => 1 } ); $dbh->{LongReadLen} = $self->{longreadlen} if ($self->{longreadlen}); $dbh->{LongTruncOk} = $self->{longtruncok} if (defined $self->{longtruncok}); # Check for connection failure if (!$dbh) { $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); } # Use consistent reads for concurrent dumping... if ($self->{debug} && !$self->{quiet}) { $self->logit("Isolation level: $self->{transaction}\n", 1); } my $sth = $dbh->prepare($self->{transaction}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->finish; # Set the date format #$dbh->do("SET LANGUAGE us_english") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $dbh->do("SET DATEFORMAT dmy") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); # Force execution of initial command $self->_ora_initial_command($dbh); # Instruct Ora2Pg that the database engine is mssql $self->{is_mssql} = 1; return $dbh; } sub _get_version { my $self = shift; my $dbver = ''; my $sql = "SELECT \@\@VERSION"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $dbver = $row[0]; last; } $sth->finish(); $dbver =~ s/ \- .*//; $dbver =~ s/\s+/ /gs; return $dbver; } sub _schema_list { my $self = shift; my $sql = qq{ SELECT s.name as schema_name, s.schema_id, u.name as schema_owner from sys.schemas s inner join sys.sysusers u on u.uid = s.principal_id WHERE s.name NOT IN ('information_schema', 'sys', 'db_accessadmin', 'db_backupoperator', 'db_datareader', 'db_datawriter', 'db_ddladmin', 'db_denydatareader', 'db_denydatawriter', 'db_owner', 'db_securityadmin') order by s.name; }; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; $sth; } sub _table_exists { my ($self, $schema, $table) = @_; my $ret = ''; my $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$schema' AND TABLE_NAME = '$table'"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $ret = $row[0]; } $sth->finish(); return $ret; } =head2 _get_encoding This function retrieves the MSSQL database encoding Returns a handle to a DB query statement. =cut sub _get_encoding { my ($self, $dbh) = @_; my $sql = qq{SELECT SERVERPROPERTY('SqlCharSetName') AS 'Instance-SqlCharSetName', SERVERPROPERTY('Collation') AS 'Instance-Collation'}; my $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my $db_encoding = ''; my $db_collation = ''; while ( my @row = $sth->fetchrow()) { $db_encoding = $row[0]; $db_collation = $row[1]; } $sth->finish(); my $db_timestamp_format = ''; my $db_date_format = ''; $sql = qq{SELECT date_format FROM sys.dm_exec_sessions WHERE session_id = \@\@spid }; $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { $db_date_format = $row[0]; } $sth->finish(); my $pg_encoding = auto_set_encoding($db_encoding); return ($db_encoding, $db_collation, $pg_encoding, $db_timestamp_format, $db_date_format); } =head2 auto_set_encoding This function is used to find the PostgreSQL charset corresponding to the MSSQL encoding valie =cut sub auto_set_encoding { my $mssql_encoding = shift; my %ENCODING = ( "iso_1" => "LATIN1", ); return $ENCODING{$mssql_encoding}; } =head2 _table_info This function retrieves all MSSQL tables information. Returns a handle to a DB query statement. =cut sub _table_info { my $self = shift; my $do_real_row_count = shift; # First register all tablespace/table in memory from this database my %tbspname = (); my $schema_clause = ''; $schema_clause = " AND s.name='$self->{schema}'" if ($self->{schema}); my $sql = qq{SELECT t.NAME AS TABLE_NAME, NULL AS comment, t.type_desc as TABLE_TYPE, p.rows AS RowCounts, SUM(a.used_pages) * 8 / 1024 AS UsedSpaceMB, CONVERT(DECIMAL,SUM(a.total_pages)) * 8 / 1024 AS TotalSpaceMB, s.Name AS TABLE_SCHEMA, SCHEMA_NAME(t.principal_id), i.type_desc, p.data_compression_desc FROM sys.tables t INNER JOIN sys.indexes i ON t.OBJECT_ID = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE t.is_ms_shipped = 0 AND i.OBJECT_ID > 255 AND t.type='U' AND t.NAME NOT LIKE '#%' $schema_clause }; my %tables_infos = (); my %comments = (); $sql .= $self->limit_to_objects('TABLE', 't.Name'); $sql .= " GROUP BY t.type_desc, i.type_desc, s.Name, t.Name, SCHEMA_NAME(t.principal_id), p.Rows, p.data_compression_desc ORDER BY s.Name, t.Name"; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[6].$row->[0]"; } $row->[2] =~ s/^USER_//; $comments{$row->[0]}{comment} = $row->[1] || ''; $comments{$row->[0]}{table_type} = $row->[2] || ''; $tables_infos{$row->[0]}{owner} = $row->[7] || $row->[6]; $tables_infos{$row->[0]}{num_rows} = $row->[3] || 0; $tables_infos{$row->[0]}{comment} = ''; # SQL Server doesn't have COMMENT and we don't play with "Extended Properties" $tables_infos{$row->[0]}{type} = $row->[2] || ''; $tables_infos{$row->[0]}{nested} = 'NO'; $tables_infos{$row->[0]}{size} = sprintf("%.3f", $row->[5]) || 0; $tables_infos{$row->[0]}{tablespace} = 0; $tables_infos{$row->[0]}{auto_increment} = 0; $tables_infos{$row->[0]}{tablespace} = $tbspname{$row->[0]} || ''; $tables_infos{$row->[0]}{partitioned} = 1 if (exists $self->{partitions_list}{"\L$row->[0]\E"}); $tables_infos{$row->[0]}{index_type} = $row->[8] if ($row->[8] =~ /^(XML|SPATIAL|.*COLUMNSTORE)$/i); $tables_infos{$row->[0]}{compressed} = $row->[9]; } $sth->finish(); if ($do_real_row_count) { foreach my $t (keys %tables_infos) { $self->logit("DEBUG: looking for real row count for table $t (aka using count(*))...\n", 1); my $tbname = "[$t]"; $tbname =~ s/\./\].\[/; $sql = "SELECT COUNT(*) FROM $tbname"; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $size = $sth->fetch(); $sth->finish(); $tables_infos{$t}{num_rows} = $size->[0]; } } return %tables_infos; } sub _column_comments { my ($self, $table) = @_; return ; # SQL Server doesn't have COMMENT and we don't play with "Extended Properties" } sub _column_info { my ($self, $table, $owner, $objtype, $recurs, @expanded_views) = @_; $objtype ||= 'TABLE'; my $condition = ''; if ($self->{schema}) { $condition .= "AND s.name='$self->{schema}' "; } $condition .= "AND tb.name='$table' " if ($table); if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'tb.name'); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/^\s*AND\s/ WHERE /; my $str = qq{SELECT c.name 'Column Name', t.Name 'Data type', c.max_length 'Max Length', c.is_nullable, object_definition(c.default_object_id), c.precision , c.scale , '', tb.name, s.name, '', c.column_id, NULL as AUTO_INCREMENT, NULL AS ENUM_INFO, object_definition(c.rule_object_id), t.is_user_defined, m.is_masked, m.masking_function FROM sys.columns c INNER JOIN sys.types t ON t.user_type_id = c.user_type_id INNER JOIN sys.tables AS tb ON tb.object_id = c.object_id INNER JOIN sys.schemas AS s ON s.schema_id = tb.schema_id LEFT JOIN sys.masked_columns AS m ON m.object_id = tb.object_id AND m.column_id = c.column_id $condition ORDER BY c.column_id}; my $sth = $self->{dbh}->prepare($str); if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); # Expected columns information stored in hash # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,ENUM_INFO my %data = (); my $pos = 0; while (my $row = $sth->fetch) { next if ($self->{drop_rowversion} && ($row->[1] eq 'rowversion' || $row->[1] eq 'timestamp')); if (!$self->{schema} && $self->{export_schema}) { $row->[8] = "$row->[9].$row->[8]"; } if (!$row->[15]) { if ($row->[4]) { $row->[4] =~ s/\s*CREATE\s+DEFAULT\s+.*\s+AS\s*//is; $row->[4] =~ s/[\[\]]+//g; } if ($row->[14]) { $row->[14] =~ s/[\[\]]+//g; $row->[14] =~ s/\s*CREATE\s+RULE\s+.*\s+AS\s*//is; $row->[14] =~ s/\@[a-z0-1_\$\#]+/VALUE/igs; $row->[14] = " CHECK ($row->[14])"; $row->[14] =~ s/[\r\n]+/ /gs; } } push(@{$data{"$row->[8]"}{"$row->[0]"}}, @$row); $pos++; } return %data; } sub _get_indexes { my ($self, $table, $owner, $generated_indexes) = @_; my $condition = ''; $condition .= "AND OBJECT_NAME(Id.object_id, DB_ID())='$table' " if ($table); if ($owner) { $condition .= "AND s.name='$owner' "; } else { $condition .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "OBJECT_NAME(Id.object_id, DB_ID())|Id.NAME"); } else { @{$self->{query_bind_params}} = (); } # When comparing number of index we need to retrieve generated index (mostly PK) my $generated = ''; my $col_generated = "''"; if ($self->{db_version} !~ /SQL Server 201[0-6]/) { $generated = " AND Id.auto_created = 0" if (!$generated_indexes); $col_generated = 'Id.auto_created'; } my $t0 = Benchmark->new; my $sth = ''; my $sql = qq{SELECT Id.name AS index_name, AC.name AS column_name, Id.is_unique AS UNIQUENESS, AC.column_id AS COLUMN_POSITION, Id.type AS INDEX_TYPE, 'U' AS TABLE_TYPE, $col_generated AS GENERATED, NULL AS JOIN_INDEX, t.name AS TABLE_NAME, s.name as TABLE_SCHEMA, Id.data_space_id AS TABLESPACE_NAME, Id.type_desc AS ITYP_NAME, Id.filter_definition AS PARAMETERS, IC.is_descending_key AS DESCEND, id.is_primary_key PRIMARY_KEY, typ.name AS COL_TYPE_NAME, IC.is_included_column FROM sys.tables AS T INNER JOIN sys.indexes Id ON T.object_id = Id.object_id INNER JOIN sys.index_columns IC ON Id.object_id = IC.object_id AND Id.index_id = IC.index_id INNER JOIN sys.all_columns AC ON IC.object_id = AC.object_id AND IC.column_id = AC.column_id INNER JOIN sys.types typ ON typ.user_type_id = AC.user_type_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE T.is_ms_shipped = 0 $generated $condition ORDER BY T.name, Id.index_id, IC.key_ordinal }; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); my %unique = (); my %idx_type = (); my %index_tablespace = (); my $nidx = 0; while (my $row = $sth->fetch) { next if ($self->{drop_rowversion} && ($row->[15] eq 'rowversion' || $row->[15] eq 'timestamp')); # Handle case where indexes name include the schema at create time $row->[0] =~ s/^$self->{schema}\.//i if ($self->{schema}); next if (!$row->[0]); my $save_tb = $row->[8]; if (!$self->{schema} && $self->{export_schema}) { $row->[8] = "$row->[9].$row->[8]"; } next if (!$self->is_in_struct($row->[8], $row->[1])); # Show a warning when an index has the same name as the table if ( !$self->{indexes_renaming} && !$self->{indexes_suffix} && (lc($row->[0]) eq lc($table)) ) { print STDERR "WARNING: index $row->[0] has the same name as the table itself. Please rename it before export or enable INDEXES_RENAMING.\n"; } # Save original column name my $colname = $row->[1]; # Quote column with unsupported symbols $row->[1] = $self->quote_object_name($row->[1]); # Covered columns (include) if ($row->[16] == 1) { push(@{$idx_type{$row->[8]}{$row->[0]}{type_include}}, $row->[1]); next; } $unique{$row->[8]}{$row->[0]} = $row->[2]; # Replace function based index type if ( $row->[13] ) { # Append DESC sort order when not default to ASC if ($row->[13] eq 'DESC') { $row->[1] .= " DESC"; } } $idx_type{$row->[8]}{$row->[0]}{type_name} = $row->[11]; $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4]; push(@{$data{$row->[8]}{$row->[0]}}, $row->[1]); $index_tablespace{$row->[8]}{$row->[0]} = $row->[10]; $nidx++; } $sth->finish(); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Collecting $nidx indexes in sys.indexes took: " . timestr($td) . "\n", 1); return \%unique, \%data, \%idx_type, \%index_tablespace; } sub _count_indexes { my ($self, $table, $owner) = @_; my $condition = ''; $condition = " FROM $self->{schema}" if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/ AND / WHERE /; my %tables_infos = (); if ($table) { $tables_infos{$table} = 1; } else { %tables_infos = Ora2Pg::MSSQL::_table_info($self); } my %data = (); # Retrieve all indexes for the given table foreach my $t (keys %tables_infos) { my $sql = "SHOW INDEX FROM `$t` $condition"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $i = 1; while (my $row = $sth->fetch) { push(@{$data{$row->[0]}{$row->[2]}}, $row->[4]); } } return \%data; } sub _foreign_key { my ($self, $table, $owner) = @_; my $condition = ''; $condition .= " AND OBJECT_NAME (f.referenced_object_id) = '$table' " if ($table); $condition .= " AND SCHEMA_NAME(t_parent.schema_id) = '$self->{schema}' " if ($self->{schema}); $condition =~ s/^ AND / WHERE /; my $deferrable = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "DEFERRABLE"; my $sql = qq{SELECT fk.name AS ConsName, SCHEMA_NAME(fk.schema_id) SchemaName, c_parent.name AS ParentColumnName, t_parent.name AS ParentTableName, t_child.name AS ReferencedTableName, c_child.name AS ReferencedColumnName, update_referential_action_desc UPDATE_RULE, delete_referential_action_desc DELETE_RULE, SCHEMA_NAME(t_parent.schema_id) FROM sys.foreign_keys fk INNER JOIN sys.foreign_key_columns fkc ON fkc.constraint_object_id = fk.object_id INNER JOIN sys.tables t_parent ON t_parent.object_id = fk.parent_object_id INNER JOIN sys.columns c_parent ON fkc.parent_column_id = c_parent.column_id AND c_parent.object_id = t_parent.object_id INNER JOIN sys.tables t_child ON t_child.object_id = fk.referenced_object_id INNER JOIN sys.columns c_child ON c_child.object_id = t_child.object_id AND fkc.referenced_column_id = c_child.column_id $condition ORDER BY t_parent.name, c_parent.name}; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); my @cons_columns = (); my $i = 1; my %data = (); my %link = (); while (my $r = $sth->fetch) { my $key_name = $r->[3] . '_' . $r->[2] . '_fk' . $i; if ($r->[0]) { $key_name = uc($r->[0]); } if (!$self->{schema} && $self->{export_schema}) { $r->[3] = "$r->[1].$r->[3]"; $r->[4] = "$r->[8].$r->[4]"; } push(@{$link{$r->[3]}{$key_name}{local}}, $r->[2]); push(@{$link{$r->[3]}{$key_name}{remote}{$r->[4]}}, $r->[5]); #SELECT ConsName, SchemaName, ColName, TableName, ReferencedTableName, ReferencedColumnName, UPDATE_RULE, DELETE_RULE, SCHEMA_NAME $r->[7] =~ s/_/ /; push(@{$data{$r->[3]}}, [ ($key_name, $key_name, '', $r->[7], 'DEFERRABLE', 'Y', '', $r->[3], '', $r->[6]) ]); $i++; } $sth->finish(); return \%link, \%data; } =head2 _get_views This function implements an MSSQL-native views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_views { my ($self) = @_; my $condition = ''; $condition .= "AND TABLE_SCHEMA='$self->{schema}' " if ($self->{schema}); my %comments = (); # Retrieve all views my $str = qq{select v.name as view_name, schema_name(v.schema_id) as schema_name, m.definition, v.with_check_option from sys.views v join sys.sql_modules m on m.object_id = v.object_id WHERE NOT EXISTS (SELECT 1 FROM sys.indexes i WHERE i.object_id = v.object_id and i.index_id = 1 and i.ignore_dup_key = 0) AND is_date_correlation_view=0}; if (!$self->{schema}) { $str .= " AND schema_name(v.schema_id) NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND schema_name(v.schema_id) = '$self->{schema}'"; } $str .= $self->limit_to_objects('VIEW', 'v.name'); $str .= " ORDER BY schema_name, view_name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %ordered_view = (); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } $row->[2] =~ s/ //g; $row->[2] =~ s/[\[\]]//g; $row->[2] =~ s/^.*\bCREATE VIEW\s+[^\s]+\s+AS\s+//is; $data{$row->[0]}{text} = $row->[2]; $data{$row->[0]}{owner} = ''; $data{$row->[0]}{comment} = ''; $data{$row->[0]}{check_option} = $row->[3]; $data{$row->[0]}{updatable} = 'Y'; $data{$row->[0]}{definer} = ''; $data{$row->[0]}{security} = ''; if ($self->{plsql_pgsql}) { $data{$row->[0]}{text} =~ s/\s+WITH\s+(ENCRYPTION|SCHEMABINDING|VIEW_METADATA)\s+AS\s+//is; $data{$row->[0]}{text} =~ s/^\s*AS\s+//is; } } return %data; } sub _get_triggers { my($self) = @_; my $str = qq{SELECT o.name AS trigger_name ,USER_NAME(o.uid) AS trigger_owner ,OBJECT_NAME(o.parent_obj) AS table_name ,s.name AS table_schema ,OBJECTPROPERTY( o.id, 'ExecIsAfterTrigger') AS isafter ,OBJECTPROPERTY( o.id, 'ExecIsInsertTrigger') AS isinsert ,OBJECTPROPERTY( o.id, 'ExecIsUpdateTrigger') AS isupdate ,OBJECTPROPERTY( o.id, 'ExecIsDeleteTrigger') AS isdelete ,OBJECTPROPERTY( o.id, 'ExecIsInsteadOfTrigger') AS isinsteadof ,OBJECTPROPERTY( o.id, 'ExecIsTriggerDisabled') AS [disabled] , c.text FROM sys.sysobjects o INNER JOIN sys.syscomments AS c ON o.id = c.id INNER JOIN sys.tables t ON o.parent_obj = t.object_id INNER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE o.type = 'TR' }; if ($self->{schema}) { $str .= " AND s.name = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','t.name|t.name|o.name'); $str .= " ORDER BY t.name, o.name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @triggers = (); while (my $row = $sth->fetch) { $row->[4] = 'AFTER'; # only FOR=AFTER trigger in this field, no BEFORE $row->[4] = 'INSTEAD OF' if ($row->[8]); my @actions = (); push(@actions, 'INSERT') if ($row->[5]); push(@actions, 'UPDATE') if ($row->[6]); push(@actions, 'DELETE') if ($row->[7]); my $act = join(' OR ', @actions); if (!$self->{schema} && $self->{export_schema}) { $row->[2] = "$row->[3].$row->[2]"; } $row->[10] =~ s/ //g; $row->[10] =~ s/^(?:.*?)\sAS\s(.*)\s*;\s*$/$1/is; push(@triggers, [ ($row->[0], $row->[4], $act, $row->[2], $row->[10], '', 'ROW', $row->[1]) ]); } return \@triggers; } sub _unique_key { my($self, $table, $owner) = @_; my %result = (); my @accepted_constraint_types = (); push @accepted_constraint_types, "'P'" unless($self->{skip_pkeys}); push @accepted_constraint_types, "'U'" unless($self->{skip_ukeys}); return %result unless(@accepted_constraint_types); my $condition = ''; $condition .= " AND t.name = '$table' " if ($table); $condition .= " AND sh.name = '$self->{schema}' " if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "t.name|i.name"); } else { @{$self->{query_bind_params}} = (); } my $sql = qq{SELECT sh.name AS schema_name, i.name AS constraint_name, t.name AS table_name, c.name AS column_name, ic.key_ordinal AS column_position, ic.is_descending_key AS is_desc, i.is_unique_constraint AS unique_key, i.is_primary_key AS primary_key, typ.name FROM sys.indexes i INNER JOIN sys.index_columns ic ON i.index_id = ic.index_id AND i.object_id = ic.object_id INNER JOIN sys.tables AS t ON t.object_id = i.object_id INNER JOIN sys.columns c ON t.object_id = c.object_id AND ic.column_id = c.column_id INNER JOIN sys.types typ ON typ.user_type_id = c.user_type_id INNER JOIN sys.objects AS syso ON syso.object_id = t.object_id AND syso.is_ms_shipped = 0 INNER JOIN sys.schemas AS sh ON sh.schema_id = t.schema_id WHERE (i.is_unique_constraint = 1 OR i.is_primary_key = 1) $condition ORDER BY sh.name, i.name, ic.key_ordinal; }; my %tables_infos = (); if ($table) { $tables_infos{$table} = 1; } else { %tables_infos = Ora2Pg::MSSQL::_table_info($self); } my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $i = 1; while (my $row = $sth->fetch) { next if ($self->{drop_rowversion} && ($row->[9] eq 'rowversion' || $row->[9] eq 'timestamp')); my $name = $row->[2]; if (!$self->{schema} && $self->{export_schema}) { $name = "$row->[0].$row->[2]"; } my $idxname = $row->[3] . '_idx' . $i; $idxname = $row->[1] if ($row->[1]); my $key_type = 'U'; $key_type = 'P' if ($row->[7]); next if (!grep(/$key_type/, @accepted_constraint_types)); if (!exists $result{$name}{$idxname}) { my %constraint = (type => $key_type, 'generated' => 'N', 'index_name' => $idxname, columns => [ ($row->[3]) ] ); $result{$name}{$idxname} = \%constraint if ($row->[3]); $i++; } else { push(@{$result{$name}{$idxname}->{columns}}, $row->[3]); } } return %result; } sub _check_constraint { my ($self, $table, $owner) = @_; my $condition = ''; $condition .= " AND t.name = '$table' " if ($table); $condition .= " AND s.name = '$self->{schema}' " if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "t.name|i.name"); } else { @{$self->{query_bind_params}} = (); } my $sql = qq{SELECT s.name SchemaName, t.name as TableName, col.name as column_name, con.name as constraint_name, con.definition, con.is_disabled FROM sys.check_constraints con LEFT OUTER JOIN sys.objects t ON con.parent_object_id = t.object_id JOIN sys.schemas AS s ON t.schema_id = s.schema_id LEFT OUTER JOIN sys.all_columns col ON con.parent_column_id = col.column_id AND con.parent_object_id = col.object_id $condition ORDER BY SchemaName, t.Name, col.name }; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $row->[1] = "$row->[0].$row->[1]"; } $row->[4] =~ s/[\[\]]//gs; $row->[4] =~ s/^\(//s; $row->[4] =~ s/\)$//s; $data{$row->[1]}{constraint}{$row->[3]}{condition} = $row->[4]; if ($row->[5]) { $data{$row->[1]}{constraint}{$row->[3]}{validate} = 'NOT VALIDATED'; } else { $data{$row->[1]}{constraint}{$row->[3]}{validate} = 'VALIDATED'; } } return %data; } sub _get_external_tables { my ($self) = @_; # There is no external table in MSSQL return; } sub _get_directory { my ($self) = @_; # There is no external table in MSSQL return; } sub _get_functions { my $self = shift; # Retrieve all functions my $str = qq{SELECT O.name, M.definition, O.type_desc, s.name, M.null_on_null_input, M.execute_as_principal_id FROM sys.sql_modules M JOIN sys.objects O ON M.object_id=O.object_id JOIN sys.schemas AS s ON o.schema_id = s.schema_id WHERE O.type IN ('IF','TF','FN') }; if ($self->{schema}) { $str .= " AND s.name = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('FUNCTION','O.name'); $str .= " ORDER BY O.name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); while (my $row = $sth->fetch) { my $kind = 'FUNCTION'; next if ( ($kind ne $self->{type}) && ($self->{type} ne 'SHOW_REPORT') ); my $fname = $row->[0]; if ($self->{export_schema} && !$self->{schema}) { $row->[0] = "$row->[3].$row->[0]"; } $functions{"$row->[0]"}{name} = $row->[0]; $functions{"$row->[0]"}{text} = $row->[1]; $functions{"$row->[0]"}{kind} = $row->[2]; $functions{"$row->[0]"}{strict} = $row->[4]; $functions{"$row->[0]"}{security} = ($row->[5] == -2) ? 'DEFINER' : 'EXECUTER'; $functions{"$row->[0]"}{text} =~ s/ //gs; if ($self->{plsql_pgsql}) { $functions{"$row->[0]"}{text} =~ s/[\[\]]//gs; } } return \%functions; } sub _get_procedures { my $self = shift; # Retrieve all functions my $str = qq{SELECT O.name, M.definition, O.type_desc, s.name, M.null_on_null_input, M.execute_as_principal_id FROM sys.sql_modules M JOIN sys.objects O ON M.object_id=O.object_id JOIN sys.schemas AS s ON o.schema_id = s.schema_id WHERE O.type = 'P' }; if ($self->{schema}) { $str .= " AND s.name = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('PROCEDURE','O.name'); $str .= " ORDER BY O.name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); while (my $row = $sth->fetch) { my $kind = 'PROCEDURE'; next if ( ($kind ne $self->{type}) && ($self->{type} ne 'SHOW_REPORT') ); my $fname = $row->[0]; if ($self->{export_schema} && !$self->{schema}) { $row->[0] = "$row->[3].$row->[0]"; } $functions{"$row->[0]"}{name} = $row->[0]; $functions{"$row->[0]"}{text} = $row->[1]; $functions{"$row->[0]"}{kind} = $row->[2]; $functions{"$row->[0]"}{strict} = $row->[4]; $functions{"$row->[0]"}{security} = ($row->[5] == -2) ? 'DEFINER' : 'EXECUTER'; $functions{"$row->[0]"}{text} =~ s/ //gs; if ($self->{plsql_pgsql}) { $functions{"$row->[0]"}{text} =~ s/[\[\]]//gs; } } return \%functions; } sub _lookup_function { my ($self, $code, $fctname) = @_; my $type = 'functions'; $type = lc($self->{type}) . 's' if ($self->{type} eq 'FUNCTION' or $self->{type} eq 'PROCEDURE'); # Replace all double quote with single quote $code =~ s/"/'/g; # replace backquote with double quote $code =~ s/`/"/g; # Remove some unused code $code =~ s/\s+READS SQL DATA//igs; $code =~ s/\s+UNSIGNED\b((?:.*?)\bFUNCTION\b)/$1/igs; while ($code =~ s/(\s*DECLARE\s+)([^\r\n]+?),\s*\@/$1 $2\n$1 \@/is) {}; my %fct_detail = (); $fct_detail{func_ret_type} = 'OPAQUE'; my @code = (); # Split data into declarative and code part ($fct_detail{declare}, @code) = split(/\b(BEGIN|SET|SELECT|INSERT|UPDATE|IF)\b/i, $code, 3); $fct_detail{code} = join(' ', @code); return if (!$fct_detail{code}); # Look for table variables in code and rewrite them as temporary tables my $records = ''; while ($fct_detail{code} =~ s/DECLARE\s+\@([^\s]+)\s+TABLE\s+(\(.*?[\)\w]\s*\))\s*([^,])/"CREATE TEMPORARY TABLE v_$1 $2" . (($3 eq ")") ? $3 : "") . ";"/eis) { my $varname = $1; $fct_detail{code} =~ s/\@$varname\b/v_$varname/igs; } # Move all DECLARE statements found in the code into the DECLARE section my @lines = split(/\n/, $fct_detail{code}); $fct_detail{code} = ''; foreach my $l (@lines) { if ($l !~ /^\s*DECLARE\s+.*CURSOR/ && $l =~ /^\s*DECLARE\s+(.*)/i) { $fct_detail{declare} .= "\n$1"; $fct_detail{declare} .= ";" if ($1 !~ /;$/); } else { $fct_detail{code} .= "$l\n"; } } # Fix DECLARE section $fct_detail{declare} =~ s/\bDECLARE\s+//igs; if ($fct_detail{declare} !~ /\bDECLARE\b/i) { if ($fct_detail{declare} !~ s/(FUNCTION|PROCEDURE|PROC)\s+([^\s\(]+)[\)\s]+AS\s+(.*)/$1 $2\nDECLARE\n$3/is) { $fct_detail{declare} =~ s/(FUNCTION|PROCEDURE|PROC)\s+([^\s\(]+)\s+(.*\@.*?[\)\s]+)(RETURNS|AS)\s+(.*)/$1 $2 ($3)\n$4\nDECLARE\n$5/is; } } # Remove any label that was before the main BEGIN block $fct_detail{declare} =~ s/\s+[^\s\:]+:\s*$//gs; $fct_detail{declare} =~ s/(RETURNS.*TABLE.*\))\s*\)\s*AS\b/) $1 AS/is; @{$fct_detail{param_types}} = (); if ( ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE|PROC)\s+([^\s]+)\s+((?:RETURNS|AS)\s+.*)//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE|PROC)\s+([^\s\(]+)(.*?)\s+((?:RETURNS|AS)\s+.*)//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE|PROC)\s+(.*?)\s+((?:RETURNS|AS)\s+.*)//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE|PROC)\s+([^\s\(]+)\s*(\(.*\))//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE|PROC)\s+([^\s\(]+)\s*DECLARE/DECLARE/is) ) { $fct_detail{before} = $1; $fct_detail{type} = uc($2); $fct_detail{name} = $3; $fct_detail{args} = $4; my $tmp_returned = $5; if ($fct_detail{args} !~ /^\(/ && !$tmp_returned) { $tmp_returned = $fct_detail{args}; $fct_detail{args} = ''; } $fct_detail{type} = 'PROCEDURE' if ($fct_detail{type} eq 'PROC'); $type = lc($fct_detail{type} . 's'); $tmp_returned =~ s/RETURNS\s+DECLARE/RETURNS /is; if ($tmp_returned =~ s/\s+AS\s+(DECLARE.*)//is) { $fct_detail{declare} .= "$1\n"; } $tmp_returned =~ s/RETURNS\s+(.*)\s+AS\s+.*/$1/is; if ($fct_detail{args} =~ s/\b(AS|DECLARE)\s+(.*)//is) { $tmp_returned = "DECLARE\n$1"; } chomp($tmp_returned); $tmp_returned =~ s/[\)\s]AS\s+.*//is; $fct_detail{code} = "\n" . $fct_detail{code}; $tmp_returned =~ s/\)\)$/\)/; $tmp_returned =~ s/\(MAX\)$//i; $fct_detail{args} =~ s/^\s*\(\s*\((.*)\)\s*\)$/$1/s; $fct_detail{args} =~ s/^\s*\(\s*(.*)\s*\)$/$1/s; #$fct_detail{code} =~ s/^DECLARE\b//is; if ($fct_detail{declare} =~ s/\s*COMMENT\s+(\?TEXTVALUE\d+\?|'[^\']+')//) { $fct_detail{comment} = $1; } $fct_detail{immutable} = 1 if ($fct_detail{declare} =~ s/\s*\bDETERMINISTIC\b//is); $fct_detail{before} = ''; # There is only garbage for the moment $fct_detail{name} =~ s/['"]//g; $fct_detail{fct_name} = $fct_detail{name}; if (!$fct_detail{args}) { $fct_detail{args} = '()'; } $fct_detail{immutable} = 1 if ($fct_detail{return} =~ s/\s*\bDETERMINISTIC\b//is); $fct_detail{immutable} = 1 if ($tmp_returned =~ s/\s*\bDETERMINISTIC\b//is); $tmp_returned =~ s/^\s+//; $tmp_returned =~ s/\s+$//; $fctname = $fct_detail{name} || $fctname; if ($type eq 'functions' && exists $self->{$type}{$fctname}{return} && $self->{$type}{$fctname}{return}) { $fct_detail{hasreturn} = 1; $fct_detail{func_ret_type} = $self->_sql_type($self->{$type}{$fctname}{return}); } elsif ($type eq 'functions' && !exists $self->{$type}{$fctname}{return} && $tmp_returned) { $tmp_returned =~ s/\s+CHARSET.*//is; #$fct_detail{func_ret_type} = $self->_sql_type($tmp_returned); $fct_detail{func_ret_type} = replace_sql_type($self, $tmp_returned); $fct_detail{hasreturn} = 1; } $fct_detail{language} = $self->{$type}{$fctname}{language}; $fct_detail{immutable} = 1 if ($self->{$type}{$fctname}{immutable} eq 'YES'); $fct_detail{security} = $self->{$type}{$fctname}{security}; if ($fct_detail{func_ret_type} =~ s/RETURNS\s+\@(.*?)\s+TABLE/TABLE/is) { $fct_detail{declare} .= "v_$1 record;\n"; } $fct_detail{func_ret_type} =~ s/RETURNS\s*//is; # Procedure that have out parameters are functions with PG if ($type eq 'procedures' && $fct_detail{args} =~ /\b(OUT|INOUT)\b/) { # set return type to empty to avoid returning void later $fct_detail{func_ret_type} = ' '; } # IN OUT should be INOUT $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; # Move the DECLARE statement from code to the declare section. #$fct_detail{declare} = ''; while ($fct_detail{code} =~ s/DECLARE\s+([^;\n\r]+)//is) { my $var = $1; $fct_detail{declare} .= "\n$var" if ($fct_detail{declare} !~ /v_$var /is); } # Rename arguments with @ replaced by p_ ($fct_detail{args}, $fct_detail{declare}, $fct_detail{code}) = replace_mssql_params($self, $fct_detail{args}, $fct_detail{declare}, $fct_detail{code}); # Now convert types if ($fct_detail{args}) { $fct_detail{args} = replace_sql_type($self, $fct_detail{args}); } if ($fct_detail{declare}) { $fct_detail{declare} = replace_sql_type($self, $fct_detail{declare}); } $fct_detail{args} =~ s/\s+/ /gs; push(@{$fct_detail{param_types}}, split(/\s*,\s*/, $fct_detail{args})); # Store type used in parameter list to lookup later for custom types map { s/^\(//; } @{$fct_detail{param_types}}; map { s/\)$//; } @{$fct_detail{param_types}}; map { s/\%ORA2PG_COMMENT\d+\%//gs; } @{$fct_detail{param_types}}; map { s/^\s*[^\s]+\s+(IN|OUT|INOUT)/$1/i; s/^((?:IN|OUT|INOUT)\s+[^\s]+)\s+[^\s]*$/$1/i; s/\(.*//; s/\s*\)\s*$//; s/\s+$//; } @{$fct_detail{param_types}}; } else { delete $fct_detail{func_ret_type}; delete $fct_detail{declare}; $fct_detail{code} = $code; } # Mark the function as having out parameters if any my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; my $nbout = $#nout+1 + $#ninout+1; $fct_detail{inout} = 1 if ($nbout > 0); # Append TABLE declaration to the declare section $fct_detail{declare} .= "\n$records" if ($records); # Rename variables with @ replaced by v_ ($fct_detail{code}, $fct_detail{declare}) = replace_mssql_variables($self, $fct_detail{code}, $fct_detail{declare}); $fct_detail{args} =~ s/\s*$//s; $fct_detail{args} =~ s/^\s*//s; $fct_detail{args} =~ s/[\s]+=[\s]+/ DEFAULT /gs; $fct_detail{code} =~ s/^[\r\n]*/\n/s; if ($fct_detail{code} !~ /\sEND$/s) { $fct_detail{code} =~ s/$/\nEND/; } # Remove %ROWTYPE from return type $fct_detail{func_ret_type} =~ s/\%ROWTYPE//igs; return %fct_detail; } sub replace_mssql_params { my ($self, $args, $declare, $code) = @_; if ($args =~ s/\s+(?:DECLARE|AS)\s+(.*)//is) { $declare .= "\n$1"; } while ($args =~ s/\@([^\s]+)\b/p_$1/s) { my $p = $1; $code =~ s/\@$p\b/p_$p/gis; } return ($args, $declare, $code); } sub replace_mssql_variables { my ($self, $code, $declare) = @_; # Look for mssql global variables and add them to the custom variable list while ($code =~ s/\b(?:SET\s+)?\@\@(?:SESSION\.)?([^\s:=]+)\s*:=\s*([^;]+);/PERFORM set_config('$2', $2, false);/is) { my $n = $1; my $v = $2; $self->{global_variables}{$n}{name} = lc($n); # Try to set a default type for the variable $self->{global_variables}{$n}{type} = 'bigint'; if ($v =~ /'[^\']*'/) { $self->{global_variables}{$n}{type} = 'varchar'; } if ($n =~ /datetime/i) { $self->{global_variables}{$n}{type} = 'timestamp'; } elsif ($n =~ /time/i) { $self->{global_variables}{$n}{type} = 'time'; } elsif ($n =~ /date/i) { $self->{global_variables}{$n}{type} = 'date'; } } my @to_be_replaced = (); # Look for local variable definition and append them to the declare section while ($code =~ s/SET\s+\@([^\s=]+)\s*=\s*/v_$1 := /is) { my $n = $1; push(@to_be_replaced, $n); } # Look for local variable definition and append them to the declare section while ($code =~ s/(^|[^\@])\@([^\s:=,]+)\s*:=\s*(.*)/$1v_$2 := $3/is) { my $n = $2; my $v = $3; # Try to set a default type for the variable my $type = 'integer'; $type = 'varchar' if ($v =~ /'[^']*'/); if ($n =~ /datetime/i) { $type = 'timestamp'; } elsif ($n =~ /time/i) { $type = 'time'; } elsif ($n =~ /date/i) { $type = 'date'; } $declare .= "v_$n $type;\n" if ($declare !~ /\b$n $type;/s); push(@to_be_replaced, $n); } # Fix other call to the same variable in the code foreach my $n (@to_be_replaced) { $code =~ s/\@$n\b/v_$n/gs; } # Look for variable definition in DECLARE section and rename them in the code too while ($declare =~ s/(^|[^\@])\@([a-z0-9_]+)/$1v_$2/is) { my $n = $2; # Fix other call to the same variable in the code $code =~ s/\@$n\b/v_$n/gs; } # Look for some global variable definition and append them to the declare section while ($code =~ /\@\@(ROWCOUNT|VERSION|LANGUAGE|SPID|MICROSOFTVERSION)/is) { my $v = uc($1); if ($v eq 'VERSION') { $code =~ s/\@\@$v/version()/igs; } elsif ($v eq 'LANGUAGE') { $code =~ s/\@\@$v/current_setting('client_encoding')/igs; } elsif ($v eq 'ROWCOUNT') { $declare .= "v_v_rowcount bigint;\n" if ($declare !~ /v_v_rowcount/s); $code =~ s/([\r\n])([^\r\n]+?)\@\@$v/\nGET DIAGNOSTICS v_v_rowcount := ROWCOUNT;\n$2 v_v_rowcount/igs; } elsif ($v eq 'SPID') { $code =~ s/\@\@$v/pg_backend_pid()/igs; } elsif ($v eq 'MICROSOFTVERSION') { $code =~ s/\@\@$v/current_setting('server_version')/igs; } } # Look for local variable definition and append them to the declare section while ($code =~ s/(^|[^\@])\@([a-z0-9_\$]+)/$1v_$2/is) { my $n = $2; next if ($n =~ /^v_/); # Try to set a default type for the variable my $type = 'varchar'; if ($n =~ /datetime/i) { $type = 'timestamp'; } elsif ($n =~ /time/i) { $type = 'time'; } elsif ($n =~ /date/i) { $type = 'date'; } $declare .= "v_$n $type;\n" if ($declare !~ /v_$n ($type|record);/is); # Fix other call to the same variable in the code $code =~ s/\@$n\b/v_$n/gs; } # Look for variable definition with SELECT statement $code =~ s/\bSET\s+([^\s=]+)\s*=\s*([^;]+\bSELECT\b[^;]+);/$1 = $2;/igs; return ($code, $declare); } sub _list_all_functions { my $self = shift; # Retrieve all functions # ROUTINE_SCHEMA | varchar(64) | NO | | | | # ROUTINE_NAME | varchar(64) | NO | | | | # ROUTINE_TYPE | varchar(9) | NO | | | | my $str = "SELECT ROUTINE_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.ROUTINES"; if ($self->{schema}) { $str .= " AND ROUTINE_SCHEMA = '$self->{schema}'"; } if ($self->{db_version} < '5.5.0') { $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; } $str .= " " . $self->limit_to_objects('FUNCTION','ROUTINE_NAME'); $str =~ s/ AND / WHERE /; $str .= " ORDER BY ROUTINE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @functions = (); while (my $row = $sth->fetch) { push(@functions, $row->[0]); } $sth->finish(); return @functions; } sub _sql_type { my ($self, $type, $len, $precision, $scale, $default, $no_blob_to_oid) = @_; my $data_type = ''; chomp($type); if ($len == -1) { return "$self->{data_type}{$type}"; } # Some length and scale may have not been extracted before if ($type =~ s/\(\s*(\d+)\s*\)//) { $len = $1; } elsif ($type =~ s/\(\s*(\d+)\s*,\s*(\d+)\s*\)//) { $len = $1; $scale = $2; } if ($type !~ /CHAR/i) { $precision = $len if (!$precision); } # Override the length if (exists $self->{data_type}{uc($type)}) { $type = uc($type); # Force uppercase $len *= 2 if ($len > 0 && $self->{double_max_varchar} && $type =~ /VARCHAR/); if ($len) { if ( $type =~ /CHAR|TEXT/ ) { # Type CHAR have default length set to 1 # Type VARCHAR(2) must have a specified length $len = 1 if (!$len && ($type eq "CHAR" || $type eq "NCHAR")); if ($self->{data_type}{$type} =~ /text/i) { return "$self->{data_type}{$type}"; } else { return "$self->{data_type}{$type}($len)"; } } elsif ($type eq 'BIT') { if ($precision > 1) { return "bit($precision)"; } else { return $self->{data_type}{$type}; } } elsif ($type =~ /(TINYINT|SMALLINT|INTEGER|BIGINT|INT|REAL|FLOAT|DECIMAL|NUMERIC|SMALLMONEY|MONEY)/i) { # This is an integer if (!$scale) { if ($precision) { if ($type =~ /(TINYINT|SMALLINT|INTEGER|BIGINT|INT)/) { if ($precision < 5) { return 'smallint'; } elsif ($precision <= 9) { return 'integer'; # The speediest in PG } else { return 'bigint'; } } elsif ($type =~ /(FLOAT|REAL)/) { return "float($precision)"; } return "numeric($precision)"; } else { # Most of the time integer should be enought? return $self->{data_type}{$type}; } } else { return "numeric($precision,$scale)"; } } $self->{use_uuid} = 1 if ($type =~ /UNIQUEIDENTIFIER/); return $self->{data_type}{$type}; } else { return $self->{data_type}{$type}; } } return $type; } sub replace_sql_type { my ($self, $str) = @_; $str =~ s/with local time zone/with time zone/igs; $str =~ s/([A-Z])ORA2PG_COMMENT/$1 ORA2PG_COMMENT/igs; $str =~ s/\(\s*MAX\s*\)//igs; # Replace type with precision my $mssqltype_regex = ''; foreach (keys %{$self->{data_type}}) { $mssqltype_regex .= quotemeta($_) . '|'; } $mssqltype_regex =~ s/\|$//; while ($str =~ /(.*)\b($mssqltype_regex)\s*\(([^\)]+)\)/i) { my $backstr = $1; my $type = uc($2); my $args = $3; if (exists $self->{data_type}{"$type($args)"}) { $str =~ s/\b$type\($args\)/$self->{data_type}{"$type($args)"}/igs; next; } if ($backstr =~ /_$/) { $str =~ s/\b($mssqltype_regex)\s*\(([^\)]+)\)/$1\%\|$2\%\|\%/is; next; } my ($precision, $scale) = split(/,/, $args); $scale ||= 0; my $len = $precision || 0; $len =~ s/\D//; if ( $type =~ /CHAR|TEXT/i ) { # Type CHAR have default length set to 1 # Type VARCHAR must have a specified length $len = 1 if (!$len && ($type eq 'CHAR' || $type eq 'NCHAR')); $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$len\%\|\%/is; } elsif ($type eq 'BIT') { if ($precision > 1) { return "bit($precision)"; } else { return $self->{data_type}{$type}; } } elsif ($precision && ($type =~ /(TINYINT|SMALLINT|INTEGER|BIGINT|INT|REAL|FLOAT|DECIMAL|NUMERIC|SMALLMONEY|MONEY)/)) { if (!$scale) { if ($type =~ /(TINYINT|SMALLINT|INTEGER|BIGINT|INT)/) { if ($self->{pg_integer_type}) { if ($precision < 5) { $str =~ s/\b$type\b\s*\([^\)]+\)/smallint/is; } elsif ($precision <= 9) { $str =~ s/\b$type\b\s*\([^\)]+\)/integer/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/bigint/is; } } else { $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$precision\%\|\%/i; } } elsif ($type =~ /(FLOAT|REAL)/) { return "float($precision)"; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$precision\%\|\%/is; } } else { $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$args\%\|\%/is; } } else { # Prevent from infinit loop $str =~ s/\(/\%\|/s; $str =~ s/\)/\%\|\%/s; } } $str =~ s/\%\|\%/\)/gs; $str =~ s/\%\|/\(/gs; # Replace datatype even without precision my %recover_type = (); my $i = 0; foreach my $type (sort { length($b) <=> length($a) } keys %{$self->{data_type}}) { while ($str =~ s/\b$type\b/%%RECOVER_TYPE$i%%/is) { $recover_type{$i} = $self->{data_type}{$type}; $i++; } } foreach $i (keys %recover_type) { $str =~ s/\%\%RECOVER_TYPE$i\%\%/$recover_type{$i}/; } if (($self->{type} eq 'COPY' || $self->{type} eq 'INSERT') && exists $SQL_TYPE{uc($str)}) { $str = $SQL_TYPE{uc($str)}; } # Set varchar without length to text $str =~ s/\bVARCHAR(\s*(?!\())/text$1/igs; return $str; } sub _get_job { my($self) = @_; # Don't work with Azure #return if ($self->{db_version} =~ /Microsoft SQL Azure/); return; # Retrieve all database job from user_jobs table my $str = qq{SELECT job.job_id, notify_level_email, name, enabled, description, step_name, command, server, database_name FROM msdb.dbo.sysjobs job INNER JOIN msdb.dbo.sysjobsteps steps ON job.job_id = steps.job_id WHERE job.enabled = 1 AND database_name = $self->{database} }; $str .= $self->limit_to_objects('JOB', 'NAME'); $str .= " ORDER BY NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $data{$row->[0]}{what} = $row->[6]; $data{$row->[0]}{interval} = $row->[0]; } return %data; } sub _get_dblink { my($self) = @_; # Don't work with Azure return if ($self->{db_version} =~ /Microsoft SQL Azure/); # Retrieve all database link from dba_db_links table my $str = qq{SELECT name, provider, data_source, catalog FROM sys.servers WHERE is_linked = 1 }; $str .= $self->limit_to_objects('DBLINK', 'name'); $str .= " ORDER BY name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { my $port = '1433'; if ($row->[2] =~ s/,(\d+)$//) { $port = $1; } $data{$row->[0]}{owner} = 'unknown'; $data{$row->[0]}{username} = 'unknown'; $data{$row->[0]}{host} = $row->[2]; $data{$row->[0]}{db} = $row->[3] || 'unknown'; $data{$row->[0]}{port} = $port; $data{$row->[0]}{backend} = $row->[1] || 'SQL Server'; } return %data; } =head2 _get_partitions This function implements an MSSQL-native partitions information. Return two hash ref with partition details and partition default. =cut sub _get_partitions { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT sch.name AS SchemaName, t.name AS TableName, i.name AS IndexName, p.partition_number, p.partition_id, i.data_space_id, f.function_id, f.type_desc, r.boundary_id, r.value AS BoundaryValue, ic.column_id AS PartitioningColumnID, c.name AS PartitioningColumnName FROM sys.tables AS t JOIN sys.indexes AS i ON t.object_id = i.object_id AND i.[type] <= 1 JOIN sys.partitions AS p ON i.object_id = p.object_id AND i.index_id = p.index_id JOIN sys.partition_schemes AS s ON i.data_space_id = s.data_space_id JOIN sys.index_columns AS ic ON ic.[object_id] = i.[object_id] AND ic.index_id = i.index_id AND ic.partition_ordinal >= 1 -- because 0 = non-partitioning column JOIN sys.columns AS c ON t.[object_id] = c.[object_id] AND ic.column_id = c.column_id JOIN sys.partition_functions AS f ON s.function_id = f.function_id LEFT JOIN sys.partition_range_values AS r ON f.function_id = r.function_id and r.boundary_id = p.partition_number LEFT OUTER JOIN sys.schemas sch ON t.schema_id = sch.schema_id }; $str .= $self->limit_to_objects('TABLE|PARTITION','t.name|t.name'); if ($self->{schema}) { $str .= " WHERE sch.name ='$self->{schema}'"; } $str .= " ORDER BY sch.name, t.name, i.name, p.partition_number, ic.column_id\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); my %default = (); my $i = 1; while (my $row = $sth->fetch) { my $tbname = $row->[1]; if ($self->{export_schema} && !$self->{schema}) { $row->[1] = "$row->[0].$row->[1]"; } #dbo | PartitionTable | PK__Partitio__357D0D3E1290FD9F | 2 | 72057594048872448 | 65601 | 65536 | RANGE | 2 | 2022-05-01 00:00:00 | 1 | col1 $parts{$row->[1]}{$row->[3]}{name} = $tbname . '_part_' . $i++; $row->[9] = "'$row->[9]'" if ($row->[9] && $row->[9] =~ /[^\d\.]/); $row->[9] = 'MAXVALUE' if ($row->[9] eq ''); push(@{$parts{$row->[1]}{$row->[3]}{info}}, { 'type' => 'RANGE', 'value' => $row->[9], 'column' => $row->[11], 'colpos' => $row->[10], 'tablespace' => '', 'owner' => ''}); } $sth->finish; return \%parts, \%default; } =head2 _get_subpartitions This function implements a MSSQL subpartitions information. Return two hash ref with partition details and partition default. =cut sub _get_subpartitions { my($self) = @_; my %subparts = (); my %default = (); # For what I know, subpartition is not supported by MSSQL return \%subparts, \%default; } =head2 _get_partitions_list This function implements a MSSQL-native partitions information. Return a hash of the partition table_name => type =cut sub _get_partitions_list { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT sch.name AS SchemaName, t.name AS TableName, i.name AS IndexName, p.partition_number, p.partition_id, i.data_space_id, f.function_id, f.type_desc, r.boundary_id, r.value AS BoundaryValue, ic.column_id AS PartitioningColumnID, c.name AS PartitioningColumnName FROM sys.tables AS t JOIN sys.indexes AS i ON t.object_id = i.object_id AND i.[type] <= 1 JOIN sys.partitions AS p ON i.object_id = p.object_id AND i.index_id = p.index_id JOIN sys.partition_schemes AS s ON i.data_space_id = s.data_space_id JOIN sys.index_columns AS ic ON ic.[object_id] = i.[object_id] AND ic.index_id = i.index_id AND ic.partition_ordinal >= 1 -- because 0 = non-partitioning column JOIN sys.columns AS c ON t.[object_id] = c.[object_id] AND ic.column_id = c.column_id JOIN sys.partition_functions AS f ON s.function_id = f.function_id LEFT JOIN sys.partition_range_values AS r ON f.function_id = r.function_id and r.boundary_id = p.partition_number LEFT OUTER JOIN sys.schemas sch ON t.schema_id = sch.schema_id }; $str .= $self->limit_to_objects('TABLE|PARTITION','t.name|t.name'); if ($self->{schema}) { $str .= " WHERE sch.name ='$self->{schema}'"; } $str .= " ORDER BY sch.name, t.name, i.name, p.partition_number, ic.column_id\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { $parts{"\L$row->[1]\E"}++; # if ($self->{export_schema} && !$self->{schema}) { # $row->[1] = "$row->[0].$row->[1]"; # } # $parts{"\L$row->[1]\E"}{count}++; # $parts{"\L$row->[1]\E"}{composite} = 0; # $parts{"\L$row->[1]\E"}{type} = 'RANGE'; # push(@{ $parts{"\L$row->[1]\E"}{columns} }, $row->[11]) if (!grep(/^$row->[11]$/, @{ $parts{"\L$row->[1]\E"}{columns} })); } $sth->finish; return %parts; } =head2 _get_partitioned_table Return a hash of the partitioned table with the number of partition =cut sub _get_partitioned_table { my ($self, %subpart) = @_; # Retrieve all partitions. my $str = qq{ SELECT sch.name AS SchemaName, t.name AS TableName, i.name AS IndexName, p.partition_number, p.partition_id, i.data_space_id, f.function_id, f.type_desc, r.boundary_id, r.value AS BoundaryValue, ic.column_id AS PartitioningColumnID, c.name AS PartitioningColumnName FROM sys.tables AS t JOIN sys.indexes AS i ON t.object_id = i.object_id AND i.[type] <= 1 JOIN sys.partitions AS p ON i.object_id = p.object_id AND i.index_id = p.index_id JOIN sys.partition_schemes AS s ON i.data_space_id = s.data_space_id JOIN sys.index_columns AS ic ON ic.[object_id] = i.[object_id] AND ic.index_id = i.index_id AND ic.partition_ordinal >= 1 -- because 0 = non-partitioning column JOIN sys.columns AS c ON t.[object_id] = c.[object_id] AND ic.column_id = c.column_id JOIN sys.partition_functions AS f ON s.function_id = f.function_id LEFT JOIN sys.partition_range_values AS r ON f.function_id = r.function_id and r.boundary_id = p.partition_number LEFT OUTER JOIN sys.schemas sch ON t.schema_id = sch.schema_id }; $str .= $self->limit_to_objects('TABLE|PARTITION','t.name|t.name'); if ($self->{schema}) { $str .= " WHERE sch.name ='$self->{schema}'"; } $str .= " ORDER BY sch.name, t.name, i.name, p.partition_number, ic.column_id\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $row->[1] = "$row->[0].$row->[1]"; } $parts{"\L$row->[1]\E"}{count}++; $parts{"\L$row->[1]\E"}{composite} = 0; $parts{"\L$row->[1]\E"}{type} = 'RANGE'; push(@{ $parts{"\L$row->[1]\E"}{columns} }, $row->[11]) if (!grep(/^$row->[11]$/, @{ $parts{"\L$row->[1]\E"}{columns} })); #dbo | PartitionTable | PK__Partitio__357D0D3E1290FD9F | 2 | 72057594048872448 | 65601 | 65536 | RANGE | 2 | 2022-05-01 00:00:00 | 1 | col1 } $sth->finish; return %parts; } =head2 _get_objects This function retrieves all object the MSSQL information =cut sub _get_objects { my $self = shift; my %infos = (); # TABLE my $sql = "SELECT t.name FROM sys.tables t INNER JOIN sys.indexes i ON t.OBJECT_ID = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE t.is_ms_shipped = 0 AND i.OBJECT_ID > 255 AND t.type='U' AND t.NAME NOT LIKE '#%'"; if (!$self->{schema}) { $sql .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql.= " AND s.name = '$self->{schema}'"; } my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{TABLE}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # VIEW $sql = "SELECT v.name from sys.views v join sys.sql_modules m on m.object_id = v.object_id WHERE NOT EXISTS (SELECT 1 FROM sys.indexes i WHERE i.object_id = v.object_id and i.index_id = 1 and i.ignore_dup_key = 0) AND is_date_correlation_view=0"; if (!$self->{schema}) { $sql .= " AND schema_name(v.schema_id) NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql.= " AND schema_name(v.schema_id) = '$self->{schema}'"; } $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{VIEW}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # TRIGGER $sql = "SELECT o.name FROM sys.sysobjects o INNER JOIN sys.tables t ON o.parent_obj = t.object_id INNER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE o.type = 'TR'"; if (!$self->{schema}) { $sql .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql.= " AND s.name = '$self->{schema}'"; } $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{TRIGGER}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # INDEX foreach my $t (@{$infos{TABLE}}) { my $auto_generated = ''; if ($self->{db_version} !~ /SQL Server 201[0-6]/) { $auto_generated = ' AND Id.auto_created = 0'; } my $sql = "SELECT Id.name AS index_name FROM sys.tables AS T INNER JOIN sys.indexes Id ON T.object_id = Id.object_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE T.is_ms_shipped = 0 AND OBJECT_NAME(Id.object_id, DB_ID())='$t' AND Id.is_primary_key = 0$auto_generated"; if (!$self->{schema}) { $sql .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql.= " AND s.name = '$self->{schema}'"; } $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my @row = $sth->fetchrow()) { next if ($row[2] eq 'PRIMARY'); push(@{$infos{INDEX}}, { ( name => $row[2], invalid => 0) }); } } # FUNCTION $sql = "SELECT O.name FROM sys.sql_modules M JOIN sys.objects O ON M.object_id=O.object_id JOIN sys.schemas AS s ON o.schema_id = s.schema_id WHERE O.type IN ('IF','TF','FN')"; if (!$self->{schema}) { $sql .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " AND s.name = '$self->{schema}'"; } $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{FUNCTION}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # PROCEDURE $sql = "SELECT O.name FROM sys.sql_modules M JOIN sys.objects O ON M.object_id=O.object_id JOIN sys.schemas AS s ON o.schema_id = s.schema_id WHERE O.type = 'P'"; if (!$self->{schema}) { $sql .= " AND s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " AND s.name = '$self->{schema}'"; } $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{PROCEDURE}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # PARTITION. $sql = qq{ SELECT sch.name AS SchemaName, t.name AS TableName, i.name AS IndexName, p.partition_number, p.partition_id, i.data_space_id, f.function_id, f.type_desc, r.boundary_id, r.value AS BoundaryValue, ic.column_id AS PartitioningColumnID, c.name AS PartitioningColumnName FROM sys.tables AS t JOIN sys.indexes AS i ON t.object_id = i.object_id AND i.[type] <= 1 JOIN sys.partitions AS p ON i.object_id = p.object_id AND i.index_id = p.index_id JOIN sys.partition_schemes AS s ON i.data_space_id = s.data_space_id JOIN sys.index_columns AS ic ON ic.[object_id] = i.[object_id] AND ic.index_id = i.index_id AND ic.partition_ordinal >= 1 -- because 0 = non-partitioning column JOIN sys.columns AS c ON t.[object_id] = c.[object_id] AND ic.column_id = c.column_id JOIN sys.partition_functions AS f ON s.function_id = f.function_id LEFT JOIN sys.partition_range_values AS r ON f.function_id = r.function_id and r.boundary_id = p.partition_number LEFT OUTER JOIN sys.schemas sch ON t.schema_id = sch.schema_id }; if ($self->{schema}) { $sql .= " WHERE sch.name ='$self->{schema}'"; } $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{'TABLE PARTITION'}}, { ( name => $row[0], invalid => 0) }); } $sth->finish; # MATERIALIZED VIEW $sql = qq{select v.name as view_name, schema_name(v.schema_id) as schema_name, i.name as index_name, m.definition from sys.views v join sys.indexes i on i.object_id = v.object_id and i.index_id = 1 and i.ignore_dup_key = 0 join sys.sql_modules m on m.object_id = v.object_id }; if (!$self->{schema}) { $sql .= " WHERE schema_name(v.schema_id) NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " WHERE schema_name(v.schema_id) = '$self->{schema}'"; } $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{'MATERIALIZED VIEW'}}, { ( name => $row[0], invalid => 0) }); } $sth->finish; return %infos; } sub _get_privilege { my($self) = @_; my %privs = (); my %roles = (); # Retrieve all privilege per table defined in this database my $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,IS_GRANTABLE FROM INFORMATION_SCHEMA.TABLE_PRIVILEGES"; if ($self->{schema}) { $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); $str .= " ORDER BY TABLE_NAME, GRANTEE"; my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved grants\n\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { # Remove the host part of the user $row->[0] =~ s/\@.*//; $row->[0] =~ s/'//g; $privs{$row->[1]}{type} = $row->[2]; if ($row->[3] eq 'YES') { $privs{$row->[1]}{grantable} = $row->[3]; } $privs{$row->[1]}{owner} = ''; push(@{$privs{$row->[1]}{privilege}{$row->[0]}}, $row->[2]); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); # Retrieve all privilege per column table defined in this database $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,COLUMN_NAME,IS_GRANTABLE FROM INFORMATION_SCHEMA.COLUMN_PRIVILEGES"; if ($self->{schema}) { $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $row->[0] =~ s/\@.*//; $row->[0] =~ s/'//g; $privs{$row->[1]}{owner} = ''; push(@{$privs{$row->[1]}{column}{$row->[3]}{$row->[0]}}, $row->[2]); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); return (\%privs, \%roles); } =head2 _get_database_size This function retrieves the size of the MSSQL database in MB =cut sub _get_database_size { my $self = shift; # Don't work with Azure return if ($self->{db_version} =~ /Microsoft SQL Azure/); my $mb_size = ''; my $condition = ''; my $sql = qq{SELECT d.name, m.size * 8 / 1024 FROM sys.master_files m JOIN sys.databases d ON d.database_id = m.database_id and m.type = 0 }; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $mb_size = sprintf("%.2f MB", $row[1]); last; } $sth->finish(); return $mb_size; } =head2 _get_largest_tables This function retrieves the list of largest table of the MSSQL database in MB =cut sub _get_largest_tables { my $self = shift; my %table_size = (); my $schema_clause = ''; $schema_clause = " AND s.name='$self->{schema}'" if ($self->{schema}); my $sql = qq{SELECT t.NAME AS TABLE_NAME, p.rows AS RowCounts, SUM(a.used_pages) * 8 / 1024 AS UsedSpaceMB, CONVERT(DECIMAL,SUM(a.total_pages)) * 8 / 1024 AS TotalSpaceMB, s.Name AS TABLE_SCHEMA FROM sys.tables t INNER JOIN sys.indexes i ON t.OBJECT_ID = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE t.is_ms_shipped = 0 AND i.OBJECT_ID > 255 AND t.type='U' $schema_clause }; $sql .= $self->limit_to_objects('TABLE', 't.Name'); $sql .= " GROUP BY t.NAME ORDER BY TotalSpaceMB"; $sql .= " LIMIT $self->{top_max}" if ($self->{top_max}); my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute(@{$self->{query_bind_params}}) or return undef; while ( my @row = $sth->fetchrow()) { $table_size{$row[0]} = $row[1]; } $sth->finish(); return %table_size; } sub _get_audit_queries { my($self) = @_; return if (!$self->{audit_user}); my @users = (); push(@users, split(/[,;\s]/, lc($self->{audit_user}))); # Retrieve all object with tablespaces. my $str = "SELECT argument FROM mysql.general_log WHERE command_type='Query' AND argument REGEXP '^(INSERT|UPDATE|DELETE|SELECT)'"; if (($#users >= 0) && !grep(/^all$/, @users)) { $str .= " AND user_host REGEXP '(" . join("'|'", @users) . ")'"; } my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved audited queries\n\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tmp_queries = (); while (my $row = $sth->fetch) { $self->_remove_comments(\$row->[0]); $row->[0] = $self->normalize_query($row->[0]); $tmp_queries{$row->[0]}++; $self->logit(".",1); } $sth->finish; $self->logit("\n", 1); my %queries = (); my $i = 1; foreach my $q (keys %tmp_queries) { $queries{$i} = $q; $i++; } return %queries; } sub _get_synonyms { my ($self) = shift; # Retrieve all synonym my $str = qq{SELECT n.name AS SchemaName, sy.name AS synonym_name, sy.base_object_name AS synonym_definition, COALESCE(PARSENAME(sy.base_object_name, 4), \@\@servername) AS server_name, COALESCE(PARSENAME(sy.base_object_name, 3), DB_NAME(DB_ID())) AS DB_name, COALESCE(PARSENAME (sy.base_object_name, 2), SCHEMA_NAME(SCHEMA_ID ())) AS schema_name, PARSENAME(sy.base_object_name, 1) AS table_name, \@\@servername AS local_server FROM sys.synonyms sy LEFT OUTER JOIN sys.schemas n ON sy.schema_id = n.schema_id }; if ($self->{schema}) { $str .= " WHERE n.name='$self->{schema}' "; } else { $str .= " WHERE n.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } $str .= $self->limit_to_objects('SYNONYM','sy.name'); my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %synonyms = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = $row->[0] . '.' . $row->[1]; } $synonyms{$row->[1]}{owner} = $row->[0]; $synonyms{$row->[1]}{table_owner} = $row->[5]; $synonyms{$row->[1]}{table_name} = $row->[6]; if ($row->[3] ne $row->[7]) { $synonyms{$row->[1]}{dblink} = $row->[3]; } } $sth->finish; return %synonyms; } sub _get_tablespaces { my ($self) = shift; return; } sub _list_tablespaces { my ($self) = shift; return; } sub _get_sequences { my ($self) = shift; my $str = qq{SELECT s.name, s.minimum_value AS minimum_value, s.maximum_value AS maximum_value, s.increment AS increment, s.current_value AS current_value, s.cache_size AS cache_size, s.is_cycling AS cycling, n.name, s.is_cached AS cached FROM sys.sequences s LEFT OUTER JOIN sys.schemas n ON s.schema_id = n.schema_id }; if (!$self->{schema}) { $str .= " WHERE n.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE n.name = '$self->{schema}'"; } $str .= $self->limit_to_objects('SEQUENCE', 's.name'); #$str .= " ORDER BY SEQUENCE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %seqs = (); while (my $row = $sth->fetch) { $row->[5] = '' if ($row->[8]); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = $row->[7] . '.' . $row->[0]; } push(@{$seqs{$row->[0]}}, @$row); } return \%seqs; } sub _extract_sequence_info { my ($self) = shift; my $str = qq{SELECT s.name, s.minimum_value AS minimum_value, s.maximum_value AS maximum_value, s.increment AS increment, s.current_value AS current_value, s.cache_size AS cache_size, s.is_cycling AS cycling, n.name, s.is_cached AS cached FROM sys.sequences s LEFT OUTER JOIN sys.schemas n ON s.schema_id = n.schema_id }; if (!$self->{schema}) { $str .= " WHERE n.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE n.name = '$self->{schema}'"; } $str .= $self->limit_to_objects('SEQUENCE', 's.name'); my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @script = (); while (my $row = $sth->fetch) { $row->[5] = '' if ($row->[8]); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = $row->[7] . '.' . $row->[0]; } my $nextvalue = $row->[4] + $row->[3]; my $alter = "ALTER SEQUENCE $self->{pg_supports_ifexists} " . $self->quote_object_name($row->[0]) . " RESTART WITH $nextvalue;"; push(@script, $alter); $self->logit("Extracted sequence information for sequence \"$row->[0]\", nextvalue: $nextvalue\n", 1); } $sth->finish(); return @script; } # MSSQL does not have sequences but we count auto_increment as sequences sub _count_sequences { my $self = shift; # Table: information_schema.tables # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # TABLE_TYPE | varchar(64) | NO | | | | # ENGINE | varchar(64) | YES | | NULL | | # VERSION | bigint(21) unsigned | YES | | NULL | | # ROW_FORMAT | varchar(10) | YES | | NULL | | # TABLE_ROWS | bigint(21) unsigned | YES | | NULL | | # AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_FREE | bigint(21) unsigned | YES | | NULL | | # AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | | # CREATE_TIME | datetime | YES | | NULL | | # UPDATE_TIME | datetime | YES | | NULL | | # CHECK_TIME | datetime | YES | | NULL | | # TABLE_COLLATION | varchar(32) | YES | | NULL | | # CHECKSUM | bigint(21) unsigned | YES | | NULL | | # CREATE_OPTIONS | varchar(255) | YES | | NULL | | # TABLE_COMMENT | varchar(2048) | NO | | | | my %seqs = (); my $sql = "SELECT TABLE_NAME, AUTO_INCREMENT FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}' AND AUTO_INCREMENT IS NOT NULL"; $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { push(@{$seqs{$row->[0]}}, @$row); } $sth->finish(); return \%seqs; } sub _column_attributes { my ($self, $table, $owner, $objtype) = @_; my $condition = ''; if ($self->{schema}) { $condition .= "AND s.name='$self->{schema}' "; } $condition .= "AND tb.name='$table' " if ($table); if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'tb.name'); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/^\s*AND\s/ WHERE /; my $sql = qq{SELECT c.name 'Column Name', c.is_nullable, object_definition(c.default_object_id), tb.name, t.Name 'Data type', c.column_id, s.name FROM sys.columns c INNER JOIN sys.types t ON t.user_type_id = c.user_type_id INNER JOIN sys.tables AS tb ON tb.object_id = c.object_id INNER JOIN sys.schemas AS s ON s.schema_id = tb.schema_id $condition ORDER BY c.column_id}; my $sth = $self->{dbh}->prepare($sql); if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { next if ($self->{drop_rowversion} && ($row->[4] eq 'rowversion' || $row->[4] eq 'timestamp')); if (!$self->{schema} && $self->{export_schema}) { $row->[3] = $row->[6] . '.' . $row->[3]; } $data{$row->[3]}{$row->[0]}{nullable} = 'N'; if ($row->[1]) { $data{$row->[3]}{$row->[0]}{nullable} = 'Y'; } $row->[2] =~ s/[\[\]]+//g; $data{$row->[3]}{$row->[0]}{default} = $row->[2]; # Store the data type of the column following its position $data{$row->[3]}{data_type}{$row->[5]} = $row->[4]; } return %data; } sub _list_triggers { my ($self) = @_; my $str = qq{SELECT o.name AS trigger_name ,OBJECT_NAME(o.parent_obj) AS table_name ,s.name AS table_schema FROM sys.sysobjects o INNER JOIN sys.tables t ON o.parent_obj = t.object_id INNER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE o.type = 'TR' }; if ($self->{schema}) { $str .= " AND s.name = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','t.name|t.name|o.name'); $str .= " ORDER BY t.name, o.name"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %triggers = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[2].$row->[1]"; } push(@{$triggers{$row->[1]}}, $row->[0]); } return %triggers; } sub _global_temp_table_info { my($self) = @_; # Useless, SQL Server has automatic removal of the GTT # when the session that create it is closed so it might # not persist. return; } sub _encrypted_columns { my ($self, $table, $owner) = @_; return; } sub _get_subpartitioned_table { my($self) = @_; return; } # Replace IF("user_status"=0,"username",NULL) # PostgreSQL (CASE WHEN "user_status"=0 THEN "username" ELSE NULL END) sub replace_if { my $str = shift; # First remove all IN (...) before processing my %in_clauses = (); my $j = 0; while ($str =~ s/\b(IN\s*\([^\(\)]+\))/,\%INCLAUSE$j\%/is) { $in_clauses{$j} = $1; $j++; } while ($str =~ s/\bIF\s*\(((?:(?!\)\s*THEN|\s*SELECT\s+|\bIF\s*\().)*)$/\%IF\%$2/is || $str =~ s/\bIF\s*\(([^\(\)]+)\)(\s+AS\s+)/(\%IF\%)$2/is) { my @if_params = (''); my $stop_learning = 0; my $idx = 1; foreach my $c (split(//, $1)) { $idx++ if (!$stop_learning && $c eq '('); $idx-- if (!$stop_learning && $c eq ')'); if ($idx == 0) { # Do not copy last parenthesis in the output string $c = '' if (!$stop_learning); # Inform the loop that we don't want to process any charater anymore $stop_learning = 1; # We have reach the end of the if() parameter # next character must be restored to the final string. $str .= $c; } elsif ($idx > 0) { # We are parsing the if() parameter part, append # the caracter to the right part of the param array. if ($c eq ',' && ($idx - 1) == 0) { # we are switching to a new parameter push(@if_params, ''); } elsif ($c ne "\n") { $if_params[-1] .= $c; } } } my $case_str = 'CASE '; for (my $i = 1; $i <= $#if_params; $i+=2) { $if_params[$i] =~ s/^\s+//gs; $if_params[$i] =~ s/\s+$//gs; if ($i < $#if_params) { if ($if_params[$i] !~ /INCLAUSE/) { $case_str .= "WHEN $if_params[0] THEN $if_params[$i] ELSE $if_params[$i+1] "; } else { $case_str .= "WHEN $if_params[0] $if_params[$i] THEN $if_params[$i+1] "; } } else { $case_str .= " ELSE $if_params[$i] "; } } $case_str .= 'END '; $str =~ s/\%IF\%/$case_str/s; } $str =~ s/\%INCLAUSE(\d+)\%/$in_clauses{$1}/gs; $str =~ s/\s*,\s*IN\s*\(/ IN \(/igs; return $str; } sub _get_plsql_metadata { my $self = shift; my $owner = shift; my $schema_clause = ''; $schema_clause = "WHERE s.name='$self->{schema}'" if ($self->{schema}); # Retrieve all functions my $str = qq{SELECT OBJECT_NAME(sm.object_id) AS object_name, SCHEMA_NAME(o.schema_id), o.type_desc, sm.definition, o.type, sm.uses_ansi_nulls, sm.uses_quoted_identifier, sm.is_schema_bound, sm.execute_as_principal_id FROM sys.sql_modules AS sm JOIN sys.objects AS o ON sm.object_id = o.object_id LEFT OUTER JOIN sys.schemas s ON o.schema_id = s.schema_id $schema_clause ORDER BY 1;}; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); while (my $row = $sth->fetch) { next if (grep(/^$row->[0]$/i, @fct_done)); push(@fct_done, "$row->[0]"); $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{type} = $row->[2]; $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{text} = $row->[3]; } $sth->finish(); # Look for functions/procedures foreach my $name (sort keys %{$self->{function_metadata}{'unknown'}{'none'}}) { # Retrieve metadata for this function after removing comments $self->_remove_comments(\$self->{function_metadata}{'unknown'}{'none'}{$name}{text}, 1); $self->{comment_values} = (); $self->{function_metadata}{'unknown'}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; my %fct_detail = $self->_lookup_function($self->{function_metadata}{'unknown'}{'none'}{$name}{text}, $name); if (!exists $fct_detail{name}) { delete $self->{function_metadata}{'unknown'}{'none'}{$name}; next; } delete $fct_detail{code}; delete $fct_detail{before}; %{$self->{function_metadata}{'unknown'}{'none'}{$name}{metadata}} = %fct_detail; delete $self->{function_metadata}{'unknown'}{'none'}{$name}{text}; } } sub _get_security_definer { my ($self, $type) = @_; # Not supported by SQL Server return; } =head2 _get_identities This function retrieve information about IDENTITY columns that must be exported as PostgreSQL serial. =cut sub _get_identities { my ($self) = @_; # Retrieve all indexes my $str = qq{SELECT s.name As SchemaName, t.name As TableName, i.name as ColumnName, i.seed_value, i.increment_value, i.last_value FROM sys.tables t JOIN sys.identity_columns i ON t.object_id=i.object_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id }; if (!$self->{schema}) { $str .= " WHERE s.name NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE s.name = '$self->{schema}'"; } $str .= $self->limit_to_objects('TABLE', 't.name'); my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %seqs = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } # GENERATION_TYPE can be ALWAYS, BY DEFAULT and BY DEFAULT ON NULL $seqs{$row->[1]}{$row->[2]}{generation} = 'BY DEFAULT'; # SEQUENCE options $row->[5] = $row->[3] || 1 if ($row->[5] eq ''); $seqs{$row->[1]}{$row->[2]}{options} = "START WITH $row->[5]"; $seqs{$row->[1]}{$row->[2]}{options} .= " INCREMENT BY $row->[4]"; $seqs{$row->[1]}{$row->[2]}{options} .= " MINVALUE $row->[3]" if ($row->[3] ne ''); # For default values don't use option at all if ( $seqs{$row->[1]}{$row->[2]}{options} eq 'START WITH 1 INCREMENT BY 1 MINVALUE 1') { delete $seqs{$row->[1]}{$row->[2]}{options}; } } return %seqs; } =head2 _get_materialized_views This function implements a mssql-native materialized views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_materialized_views { my($self) = @_; my $str = qq{select v.name as view_name, schema_name(v.schema_id) as schema_name, i.name as index_name, m.definition from sys.views v join sys.indexes i on i.object_id = v.object_id and i.index_id = 1 and i.ignore_dup_key = 0 join sys.sql_modules m on m.object_id = v.object_id }; if (!$self->{schema}) { $str .= " WHERE schema_name(v.schema_id) NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE schema_name(v.schema_id) = '$self->{schema}'"; } $str .= $self->limit_to_objects('MVIEW', 'v.name'); $str .= " ORDER BY schema_name, view_name"; my $sth = $self->{dbh}->prepare($str); if (not defined $sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } if (not $sth->execute(@{$self->{query_bind_params}})) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); return (); } my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } $row->[3] =~ s/ //g; $row->[3] =~ s/[\[\]]//g; $row->[3] =~ s/^CREATE VIEW [^\s]+//; $data{$row->[0]}{text} = $row->[3]; $data{$row->[0]}{updatable} = 0; $data{$row->[0]}{refresh_mode} = ''; $data{$row->[0]}{refresh_method} = ''; $data{$row->[0]}{no_index} = 0; $data{$row->[0]}{rewritable} = 0; $data{$row->[0]}{build_mode} = ''; $data{$row->[0]}{owner} = $row->[1]; } return %data; } sub _get_materialized_view_names { my($self) = @_; my $str = qq{select v.name as view_name, schema_name(v.schema_id) as schema_name, i.name as index_name, from sys.views v join sys.indexes i on i.object_id = v.object_id and i.index_id = 1 and i.ignore_dup_key = 0 }; if (!$self->{schema}) { $str .= " WHERE schema_name(v.schema_id) NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE schema_name(v.schema_id) = '$self->{schema}'"; } $str .= $self->limit_to_objects('MVIEW', 'v.name'); $str .= " ORDER BY schema_name, view_name"; my $sth = $self->{dbh}->prepare($str); if (not defined $sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } if (not $sth->execute(@{$self->{query_bind_params}})) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } my @data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } push(@data, uc($row->[0])); } return @data; } sub _get_package_function_list { my ($self, $owner) = @_; # not package in MSSQL return; } sub _get_types { my ($self, $name) = @_; # Retrieve all user defined types => PostgreSQL DOMAIN my $idx = 1; my $str = qq{SELECT t1.name, s.name, t2.name, t1.precision, t1.scale, t1.max_length, t1.is_nullable, object_definition(t1.default_object_id), object_definition(t1.rule_object_id), t1.is_table_type FROM sys.types t1 JOIN sys.types t2 ON t2.system_type_id = t1.system_type_id AND t2.is_user_defined = 0 LEFT OUTER JOIN sys.schemas s ON t1.schema_id = s.schema_id WHERE t1.is_user_defined = 1 AND t2.name <> 'sysname'}; if ($name) { $str .= " AND t1.name='$name'"; } if ($self->{schema}) { $str .= "AND s.name='$self->{schema}' "; } if (!$name) { $str .= $self->limit_to_objects('TYPE', 't1.name'); } else { @{$self->{query_bind_params}} = (); } $str .= " ORDER BY t1.name"; # use a separeate connection my $local_dbh = _db_connection($self); my $sth = $local_dbh->prepare($str) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); my @types = (); my @fct_done = (); while (my $row = $sth->fetch) { my $origname = $row->[0]; if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } $self->logit("\tFound Type: $row->[0]\n", 1); next if (grep(/^$row->[0]$/, @fct_done)); push(@fct_done, $row->[0]); my %tmp = (); if (!$row->[9]) { my $precision = ''; if ($row->[3]) { $precision .= "($row->[3]"; $precision .= ",$row->[4]" if ($row->[4]); } elsif ($row->[5]) { $precision .= "($row->[5]"; } $precision .= ")" if ($precision); my $notnull = ''; $notnull = ' NOT NULL' if (!$row->[6]); my $default = ''; if ($row->[7]) { $row->[7] =~ s/\s*CREATE\s+DEFAULT\s+.*\s+AS\s*//is; $default = " DEFAULT $row->[7]"; } my $rule = ''; if ($row->[8]) { $row->[8] =~ s/\s*CREATE\s+RULE\s+.*\s+AS\s*//is; $row->[8] =~ s/\@[a-z0-1_\$\#]+/VALUE/igs; $rule = " CHECK ($row->[8])"; $rule =~ s/[\r\n]+/ /gs; } $tmp{code} = "CREATE TYPE $row->[0] FROM $row->[2]$precision$notnull$default$rule;"; # Add domain type to main type convertion hash if ($self->{is_mssql} && !exists $self->{data_type}{uc($origname)} && ($self->{type} eq 'COPY' || $self->{type} eq 'INSERT') ) { $self->{data_type}{uc($origname)} = replace_sql_type($self, $row->[2]); } } $tmp{name} = $row->[0]; $tmp{owner} = $row->[1]; $tmp{pos} = $idx++; if (!$self->{preserve_case}) { $tmp{code} =~ s/(TYPE\s+)"[^"]+"\."[^"]+"/$1\L$row->[0]\E/igs; $tmp{code} =~ s/(TYPE\s+)"[^"]+"/$1\L$row->[0]\E/igs; } else { $tmp{code} =~ s/((?:CREATE|REPLACE|ALTER)\s+TYPE\s+)([^"\s]+)\s/$1"$2" /igs; } $tmp{code} =~ s/\s+ALTER/;\nALTER/igs; push(@types, \%tmp); } $sth->finish(); # Retrieve all user defined table types => PostgreSQL TYPE $str = qq{SELECT t1.name AS table_Type, s.name SchemaName, c.name AS ColName, c.column_id, y.name AS DataType, c.precision, c.scale, c.max_length, c.is_nullable, object_definition(t1.default_object_id), object_definition(t1.rule_object_id) FROM sys.table_types t1 INNER JOIN sys.columns c ON c.object_id = t1.type_table_object_id INNER JOIN sys.types y ON y.user_type_id = c.user_type_id LEFT OUTER JOIN sys.schemas s ON t1.schema_id = s.schema_id }; if ($name) { $str .= " AND t1.name='$name'"; } if ($self->{schema}) { $str .= "AND s.name='$self->{schema}' "; } if (!$name) { $str .= $self->limit_to_objects('TYPE', 't1.name'); } else { @{$self->{query_bind_params}} = (); } $str =~ s/ AND / WHERE /s; $str .= " ORDER BY t1.name, c.column_id"; $sth = $local_dbh->prepare($str) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); my $current_type = ''; my $old_type = ''; while (my $row = $sth->fetch) { next if ($self->{drop_rowversion} && ($row->[4] eq 'rowversion' || $row->[4] eq 'timestamp')); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } if ($old_type ne $row->[0]) { $self->logit("\tFound Type: $row->[0]\n", 1); if ($current_type ne '') { $current_type =~ s/,$//s; $current_type .= ");"; my %tmp = ( code => $current_type, name => $old_type, owner => '', pos => $idx++ ); if (!$self->{preserve_case}) { $tmp{code} =~ s/(TYPE\s+)"[^"]+"\."[^"]+"/$1\L$old_type\E/igs; $tmp{code} =~ s/(TYPE\s+)"[^"]+"/$1\L$old_type\E/igs; } else { $tmp{code} =~ s/((?:CREATE|REPLACE|ALTER)\s+TYPE\s+)([^"\s]+)\s/$1"$2" /igs; } $tmp{code} =~ s/\s+ALTER/;\nALTER/igs; push(@types, \%tmp); $current_type = ''; } $old_type = $row->[0]; } if ($current_type eq '') { $current_type = "CREATE TYPE $row->[0] AS OBJECT (" } my $precision = ''; if ($row->[5]) { $precision .= "($row->[5]"; $precision .= ",$row->[6]" if ($row->[6]); } elsif ($row->[7]) { $precision .= "($row->[7]"; } $precision .= ")" if ($precision); my $notnull = ''; $notnull = 'NOT NULL' if (!$row->[8]); my $default = ''; if ($row->[9]) { $row->[9] =~ s/\s*CREATE\s+DEFAULT\s+.*\s+AS\s*//is; $default = " DEFAULT $row->[9]"; } my $rule = ''; if ($row->[10]) { $row->[10] =~ s/\s*CREATE\s+RULE\s+.*\s+AS\s*//is; $row->[10] =~ s/\@[a-z0-1_\$\#]+/VALUE/igs; $rule = " CHECK ($row->[10])"; } $current_type .= "\n\t$row->[2] $row->[4]$precision $notnull$default$rule," } $sth->finish(); $local_dbh->disconnect() if ($local_dbh); # Process last table type if ($current_type ne '') { $current_type =~ s/,$//s; $current_type .= ");"; my %tmp = ( code => $current_type, name => $old_type, owner => '', pos => $idx++ ); if (!$self->{preserve_case}) { $tmp{code} =~ s/(TYPE\s+)"[^"]+"\."[^"]+"/$1\L$old_type\E/igs; $tmp{code} =~ s/(TYPE\s+)"[^"]+"/$1\L$old_type\E/igs; } else { $tmp{code} =~ s/((?:CREATE|REPLACE|ALTER)\s+TYPE\s+)([^"\s]+)\s/$1"$2" /igs; } $tmp{code} =~ s/\s+ALTER/;\nALTER/igs; push(@types, \%tmp); } return \@types; } sub _col_count { my ($self, $table, $schema) = @_; my $condition = ''; if ($schema) { $condition .= "AND s.name='$self->{schema}' "; } $condition .= "AND t.name='$table' " if ($table); if (!$table) { $condition .= $self->limit_to_objects('TABLE', 't.name'); } else { @{$self->{query_bind_params}} = (); } if ($self->{drop_rowversion}) { $condition .= "AND typ.name NOT IN ( 'rowversion', 'timestamp')"; } $condition =~ s/^\s*AND\s/ WHERE /; my $sql = qq{SELECT s.name, t.name, count(*) FROM sys.columns c INNER JOIN sys.tables AS t ON t.object_id = c.object_id INNER JOIN sys.schemas AS s ON s.schema_id = t.schema_id INNER JOIN sys.types AS typ ON c.user_type_id = typ.user_type_id $condition GROUP BY s.name, t.name}; my $sth = $self->{dbh}->prepare($sql) || $self->logit("FATAL: _col_count() " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $data{"$row->[0].$row->[1]"} = $row->[2]; } else { $data{$row->[1]} = $row->[2]; } } return %data; } 1; ora2pg-25.0/lib/Ora2Pg/MySQL.pm000066400000000000000000002602731500113072400160000ustar00rootroot00000000000000package Ora2Pg::MySQL; use vars qw($VERSION); use strict; use DBI; use POSIX qw(locale_h); #set locale to LC_NUMERIC C setlocale(LC_NUMERIC,"C"); $VERSION = '25.0'; # Some function might be excluded from export and assessment. our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); # These definitions can be overriden from configuration # file using the DATA_TYPË configuration directive. our %SQL_TYPE = ( 'TINYINT UNSIGNED' => 'smallint', 'SMALLINT UNSIGNED' => 'integer', 'MEDIUMINT UNSIGNED' => 'integer', 'BIGINT UNSIGNED' => 'numeric', 'INT UNSIGNED' => 'bigint', 'TINYINT' => 'smallint', # 1 byte 'SMALLINT' => 'smallint', # 2 bytes 'MEDIUMINT' => 'integer', # 3 bytes 'INT' => 'integer', # 4 bytes 'BIGINT' => 'bigint', # 8 bytes 'DECIMAL' => 'decimal', 'DEC' => 'decimal', 'NUMERIC' => 'numeric', 'FIXED' => 'numeric', 'FLOAT' => 'double precision', 'REAL' => 'real', 'DOUBLE PRECISION' => 'double precision', 'DOUBLE' => 'double precision', 'BOOLEAN' => 'boolean', 'BOOL' => 'boolean', 'CHAR' => 'char', 'VARCHAR' => 'varchar', 'TINYTEXT' => 'text', 'TEXT' => 'text', 'MEDIUMTEXT' => 'text', 'LONGTEXT' => 'text', 'VARBINARY' => 'bytea', 'BINARY' => 'bytea', 'TINYBLOB' => 'bytea', 'BLOB' => 'bytea', 'MEDIUMBLOB' => 'bytea', 'LONGBLOB' => 'bytea', 'ENUM' => 'text', 'SET' => 'text', 'DATE' => 'date', 'DATETIME' => 'timestamp without time zone', 'TIME' => 'time without time zone', 'TIMESTAMP' => 'timestamp without time zone', 'YEAR' => 'smallint', 'MULTIPOLYGON' => 'geometry', 'BIT' => 'bit varying', 'UNSIGNED' => 'bigint' ); sub _db_connection { my $self = shift; $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$self->{quiet}); if (!defined $self->{oracle_pwd}) { eval("use Term::ReadKey;"); if (!$@) { $self->{oracle_user} = $self->_ask_username('MySQL') unless (defined $self->{oracle_user}); $self->{oracle_pwd} = $self->_ask_password('MySQL'); } } my $dbh = DBI->connect("$self->{oracle_dsn}", $self->{oracle_user}, $self->{oracle_pwd}, { 'RaiseError' => 1, AutoInactiveDestroy => 1, mysql_enable_utf8 => 1, mysql_conn_attrs => { program_name => 'ora2pg ' || $VERSION } } ); # Check for connection failure if (!$dbh) { $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); } # Use consistent reads for concurrent dumping... #$dbh->do('START TRANSACTION WITH CONSISTENT SNAPSHOT;') || $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); if ($self->{debug} && !$self->{quiet}) { $self->logit("Isolation level: $self->{transaction}\n", 1); } my $sth = $dbh->prepare($self->{transaction}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->finish; # Get SQL_MODE from the MySQL database $sth = $dbh->prepare('SELECT @@sql_mode') or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $self->{mysql_mode} = $row->[0]; } $sth->finish; if ($self->{nls_lang}) { if ($self->{debug} && !$self->{quiet}) { $self->logit("Set default encoding to '$self->{nls_lang}' and collate to '$self->{nls_nchar}'\n", 1); } my $collate = ''; $collate = " COLLATE '$self->{nls_nchar}'" if ($self->{nls_nchar}); $sth = $dbh->prepare("SET NAMES '$self->{nls_lang}'$collate") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->finish; } # Force execution of initial command $self->_ora_initial_command($dbh); if ($self->{mysql_mode} =~ /PIPES_AS_CONCAT/) { $self->{mysql_pipes_as_concat} = 1; } # Instruct Ora2Pg that the database engine is mysql $self->{is_mysql} = 1; return $dbh; } sub _get_version { my $self = shift; my $oraver = ''; my $sql = "SELECT version()"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $oraver = $row[0]; last; } $sth->finish(); $oraver =~ s/ \- .*//; return $oraver; } sub _schema_list { my $self = shift; my $sql = "SHOW DATABASES WHERE `Database` NOT IN ('information_schema', 'performance_schema');"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; $sth; } sub _table_exists { my ($self, $schema, $table) = @_; my $ret = ''; my $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$schema' AND TABLE_NAME = '$table'"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $ret = $row[0]; } $sth->finish(); return $ret; } =head2 _get_encoding This function retrieves the Oracle database encoding Returns a handle to a DB query statement. =cut sub _get_encoding { my ($self, $dbh) = @_; my $sql = "SHOW VARIABLES LIKE 'character\\_set\\_%';"; my $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my $my_encoding = ''; my $my_client_encoding = ''; while ( my @row = $sth->fetchrow()) { if ($row[0] eq 'character_set_database') { $my_encoding = $row[1]; } elsif ($row[0] eq 'character_set_client') { $my_client_encoding = $row[1]; } } $sth->finish(); my $my_timestamp_format = ''; my $my_date_format = ''; $sql = "SHOW VARIABLES LIKE '%\\_format';"; $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { if ($row[0] eq 'datetime_format') { $my_timestamp_format = $row[1]; } elsif ($row[0] eq 'date_format') { $my_date_format = $row[1]; } } $sth->finish(); #my $pg_encoding = auto_set_encoding($charset); my $pg_encoding = $my_encoding; return ($my_encoding, $my_client_encoding, $pg_encoding, $my_timestamp_format, $my_date_format); } =head2 _table_info This function retrieves all MySQL tables information. Returns a handle to a DB query statement. =cut sub _table_info { my $self = shift; # First register all tablespace/table in memory from this database my %tbspname = (); my $sth = $self->{dbh}->prepare("SELECT DISTINCT TABLE_NAME, TABLESPACE_NAME FROM INFORMATION_SCHEMA.FILES WHERE table_schema = '$self->{schema}' AND TABLE_NAME IS NOT NULL") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $r = $sth->fetch) { $tbspname{$r->[0]} = $r->[1]; } $sth->finish(); # Table: information_schema.tables # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # TABLE_TYPE | varchar(64) | NO | | | | # ENGINE | varchar(64) | YES | | NULL | | # VERSION | bigint(21) unsigned | YES | | NULL | | # ROW_FORMAT | varchar(10) | YES | | NULL | | # TABLE_ROWS | bigint(21) unsigned | YES | | NULL | | # AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_FREE | bigint(21) unsigned | YES | | NULL | | # AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | | # CREATE_TIME | datetime | YES | | NULL | | # UPDATE_TIME | datetime | YES | | NULL | | # CHECK_TIME | datetime | YES | | NULL | | # TABLE_COLLATION | varchar(32) | YES | | NULL | | # CHECKSUM | bigint(21) unsigned | YES | | NULL | | # CREATE_OPTIONS | varchar(255) | YES | | NULL | | # TABLE_COMMENT | varchar(2048) | NO | | | | my %tables_infos = (); my %comments = (); my $sql = "SELECT TABLE_NAME,TABLE_COMMENT,TABLE_TYPE,TABLE_ROWS,ROUND( ( data_length + index_length) / 1024 / 1024, 2 ) AS \"Total Size Mb\", AUTO_INCREMENT, CREATE_OPTIONS FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}'"; $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $row->[2] =~ s/^BASE //; $comments{$row->[0]}{comment} = $row->[1]; $comments{$row->[0]}{table_type} = $row->[2]; $tables_infos{$row->[0]}{owner} = ''; $tables_infos{$row->[0]}{num_rows} = $row->[3] || 0; $tables_infos{$row->[0]}{comment} = $comments{$row->[0]}{comment} || ''; $tables_infos{$row->[0]}{type} = $comments{$row->[0]}{table_type} || ''; $tables_infos{$row->[0]}{nested} = ''; $tables_infos{$row->[0]}{size} = $row->[4] || 0; $tables_infos{$row->[0]}{tablespace} = 0; $tables_infos{$row->[0]}{auto_increment} = $row->[5] || 0; $tables_infos{$row->[0]}{tablespace} = $tbspname{$row->[0]} || ''; $tables_infos{$row->[0]}{partitioned} = ($row->[6] eq 'partitioned' || exists $self->{partitions}{$row->[0]}) ? 1 : 0; # Get creation option unavailable in information_schema if ($row->[6] eq 'FEDERATED') { my $sth2 = $self->{dbh}->prepare("SHOW CREATE TABLE `$row->[0]`") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0); $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { if ($r->[1] =~ /CONNECTION='([^']+)'/) { $tables_infos{$row->[0]}{connection} = $1; } last; } $sth2->finish(); } } $sth->finish(); return %tables_infos; } sub _column_comments { my ($self, $table) = @_; my $condition = ''; my $sql = "SELECT COLUMN_NAME,COLUMN_COMMENT,TABLE_NAME,'' AS \"Owner\" FROM INFORMATION_SCHEMA.COLUMNS"; if ($self->{schema}) { $sql .= " WHERE TABLE_SCHEMA='$self->{schema}' "; } $sql .= "AND TABLE_NAME='$table' " if ($table); if (!$table) { $sql .= $self->limit_to_objects('TABLE','TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } my $sth = $self->{dbh}->prepare($sql) or $self->logit("WARNING only: " . $self->{dbh}->errstr . "\n", 0, 0); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { next if (!$self->is_in_struct($row->[2], $row->[0])); $data{$row->[2]}{$row->[0]} = $row->[1]; } return %data; } sub _column_info { my ($self, $table, $owner, $objtype, $recurs, @expanded_views) = @_; $objtype ||= 'TABLE'; my $condition = ''; if ($self->{schema}) { $condition .= "AND TABLE_SCHEMA='$self->{schema}' "; } $condition .= "AND TABLE_NAME='$table' " if ($table); if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/^AND/WHERE/; # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # COLUMN_NAME | varchar(64) | NO | | | | # ORDINAL_POSITION | bigint(21) unsigned | NO | | 0 | | # COLUMN_DEFAULT | longtext | YES | | NULL | | # IS_NULLABLE | varchar(3) | NO | | | | # DATA_TYPE | varchar(64) | NO | | | | # CHARACTER_MAXIMUM_LENGTH | bigint(21) unsigned | YES | | NULL | | # CHARACTER_OCTET_LENGTH | bigint(21) unsigned | YES | | NULL | | # NUMERIC_PRECISION | bigint(21) unsigned | YES | | NULL | | # NUMERIC_SCALE | bigint(21) unsigned | YES | | NULL | | # CHARACTER_SET_NAME | varchar(32) | YES | | NULL | | # COLLATION_NAME | varchar(32) | YES | | NULL | | # COLUMN_TYPE | longtext | NO | | NULL | | # COLUMN_KEY | varchar(3) | NO | | | | # EXTRA | varchar(27) | NO | | | | # PRIVILEGES | varchar(80) | NO | | | | # COLUMN_COMMENT | varchar(1024) | NO | | | | # GENERATION_EXPRESSION | longtext | NO | | NULL | | my $str = qq{SELECT COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH, IS_NULLABLE, COLUMN_DEFAULT, NUMERIC_PRECISION, NUMERIC_SCALE, CHARACTER_OCTET_LENGTH, TABLE_NAME, '' AS OWNER, '' AS VIRTUAL_COLUMN, ORDINAL_POSITION, EXTRA, COLUMN_TYPE, GENERATION_EXPRESSION FROM INFORMATION_SCHEMA.COLUMNS $condition ORDER BY ORDINAL_POSITION}; if ($self->{db_version} < '5.7.0') { $str =~ s/, GENERATION_EXPRESSION//; } my $sth = $self->{dbh}->prepare($str); if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); # Expected columns information stored in hash # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,ENUM_INFO my %data = (); my $pos = 0; while (my $row = $sth->fetch) { next if (!$self->is_in_struct($row->[8], $row->[0])); $row->[4] =~ s/^_[^']+\\'(.*)\\'/'$1'/; # fix collation on string if ($row->[1] eq 'enum') { if ($self->{db_version} < '5.7.0') { $row->[1] = $row->[-1]; } else { $row->[1] = $row->[-2]; } } if ($row->[13] =~ s/(decimal.*)\s+unsigned//) { $row->[1] = $1; } elsif ($row->[13] =~ /unsigned/) { $row->[1] .= ' unsigned'; } $row->[10] = $pos; $row->[12] =~ s/\s+ENABLE//is; if ($row->[12] =~ s/\bGENERATED\s+(ALWAYS|BY\s+DEFAULT)\s+(ON\s+NULL\s+)?AS\s+IDENTITY\s*(.*)//is) { $self->{identity_info}{$row->[8]}{$row->[0]}{generation} = $1; my $options = $3; $self->{identity_info}{$row->[8]}{$row->[0]}{options} = $3; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/(SCALE|EXTEND|SESSION)_FLAG: .//isg; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/KEEP_VALUE: .//is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/(START WITH):/$1/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/(INCREMENT BY):/$1/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/MAX_VALUE:/MAXVALUE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/MIN_VALUE:/MINVALUE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/CYCLE_FLAG: N/NO CYCLE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/NOCYCLE/NO CYCLE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/CYCLE_FLAG: Y/CYCLE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/CACHE_SIZE:/CACHE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/CACHE_SIZE:/CACHE/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/ORDER_FLAG: .//is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/,//gs; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s$//s; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/CACHE\s+0/CACHE 1/is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s*NOORDER//is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s*NOKEEP//is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s*NOSCALE//is; $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s*NOT\s+NULL//is; # Be sure that we don't exceed the bigint max value, # we assume that the increment is always positive if ($self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ /MAXVALUE\s+(\d+)/is) { $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/(MAXVALUE)\s+\d+/$1 9223372036854775807/is; } $self->{identity_info}{$row->[8]}{$row->[0]}{options} =~ s/\s+/ /igs; } elsif ($row->[12] =~ s/\bGENERATED\b//is) { $row->[10] = 'YES'; $row->[14] =~ s/\`//g; $row->[4] = $row->[14]; } push(@{$data{"$row->[8]"}{"$row->[0]"}}, @$row); pop(@{$data{"$row->[8]"}{"$row->[0]"}}); $pos++; } return %data; } sub _get_indexes { my ($self, $table, $owner, $generated_indexes) = @_; my $condition = ''; $condition = " FROM $self->{schema}" if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/ AND / WHERE /; my %tables_infos = (); if ($table) { $tables_infos{$table} = 1; } else { %tables_infos = Ora2Pg::MySQL::_table_info($self); } my %data = (); my %unique = (); my %idx_type = (); my %index_tablespace = (); # Retrieve all indexes for the given table foreach my $t (keys %tables_infos) { my $sql = "SHOW INDEX FROM `$t` $condition"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $i = 1; while (my $row = $sth->fetch) { next if ($row->[2] eq 'PRIMARY'); #Table : The name of the table. #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. #Seq_in_index : The column sequence number in the index, starting with 1. #Column_name : The column name. #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). #Cardinality : An estimate of the number of unique values in the index. #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. #Packed : Indicates how the key is packed. NULL if it is not. #Null : Contains YES if the column may contain NULL values and '' if not. #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. next if (!$self->is_in_struct($row->[0], $row->[4])); my $idxname = $row->[2]; $row->[1] = 'UNIQUE' if (!$row->[1]); $unique{$row->[0]}{$idxname} = $row->[1]; # Set right label to spatial index if ($row->[10] =~ /SPATIAL/) { $row->[10] = 'SPATIAL_INDEX'; } $idx_type{$row->[0]}{$idxname}{type_name} = $row->[10]; # Save original column name my $colname = $row->[4]; # Enclose with double quote if required $row->[4] = $self->quote_object_name($row->[4]); if ($self->{preserve_case}) { if (($row->[4] !~ /".*"/) && ($row->[4] !~ /\(.*\)/)) { $row->[4] =~ s/^/"/; $row->[4] =~ s/$/"/; } } # Set the index expression if ($row->[14] ne '') { $row->[4] = $row->[14]; } # Append DESC sort order when not default to ASC if ($row->[5] eq 'D') { $row->[4] .= " DESC"; } push(@{$data{$row->[0]}{$idxname}}, $row->[4]); $index_tablespace{$row->[0]}{$idxname} = ''; } } return \%unique, \%data, \%idx_type, \%index_tablespace; } sub _count_indexes { my ($self, $table, $owner) = @_; my $condition = ''; $condition = " FROM $self->{schema}" if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/ AND / WHERE /; my %tables_infos = (); if ($table) { $tables_infos{$table} = 1; } else { %tables_infos = Ora2Pg::MySQL::_table_info($self); } my %data = (); # Retrieve all indexes for the given table foreach my $t (keys %tables_infos) { my $sql = "SHOW INDEX FROM `$t` $condition"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $i = 1; while (my $row = $sth->fetch) { #Table : The name of the table. #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. #Seq_in_index : The column sequence number in the index, starting with 1. #Column_name : The column name. #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). #Cardinality : An estimate of the number of unique values in the index. #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. #Packed : Indicates how the key is packed. NULL if it is not. #Null : Contains YES if the column may contain NULL values and '' if not. #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. push(@{$data{$row->[0]}{$row->[2]}}, $row->[4]); } } return \%data; } sub _foreign_key { my ($self, $table, $owner) = @_; my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); $condition .= "AND A.CONSTRAINT_SCHEMA='$self->{schema}' " if ($self->{schema}); my $deferrable = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "DEFERRABLE"; my $sql = "SELECT DISTINCT A.COLUMN_NAME,A.ORDINAL_POSITION,A.TABLE_NAME,A.REFERENCED_TABLE_NAME,A.REFERENCED_COLUMN_NAME,A.POSITION_IN_UNIQUE_CONSTRAINT,A.CONSTRAINT_NAME,A.REFERENCED_TABLE_SCHEMA,B.MATCH_OPTION,B.UPDATE_RULE,B.DELETE_RULE FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS A INNER JOIN INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS AS B ON A.CONSTRAINT_NAME = B.CONSTRAINT_NAME WHERE A.REFERENCED_COLUMN_NAME IS NOT NULL $condition ORDER BY A.ORDINAL_POSITION,A.POSITION_IN_UNIQUE_CONSTRAINT"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); my @cons_columns = (); my $i = 1; my %data = (); my %link = (); while (my $r = $sth->fetch) { my $key_name = $r->[2] . '_' . $r->[0] . '_fk' . $i; if ($r->[6] ne 'PRIMARY') { $key_name = uc($r->[6]); } if ($self->{schema} && (lc($r->[7]) ne lc($self->{schema}))) { print STDERR "WARNING: Foreign key $r->[2].$r->[0] point to an other database: $r->[7].$r->[3].$r->[4], please fix it.\n"; } next if (!$self->is_in_struct($r->[2], $r->[0])); next if (!$self->is_in_struct($r->[3], $r->[4])); push(@{$link{$r->[2]}{$key_name}{local}}, $r->[0]); push(@{$link{$r->[2]}{$key_name}{remote}{$r->[3]}}, $r->[4]); $r->[8] = 'SIMPLE'; # See pathetical documentation of mysql # SELECT CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE push(@{$data{$r->[2]}}, [ ($key_name, $key_name, $r->[8], $r->[10], 'DEFERRABLE', 'Y', '', $r->[2], '', $r->[9]) ]); $i++; } $sth->finish(); return \%link, \%data; } =head2 _get_views This function implements an Oracle-native views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_views { my ($self) = @_; my $condition = ''; $condition .= "AND TABLE_SCHEMA='$self->{schema}' " if ($self->{schema}); # Retrieve comment of each columns # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # VIEW_DEFINITION | longtext | NO | | NULL | | # CHECK_OPTION | varchar(8) | NO | | | | # IS_UPDATABLE | varchar(3) | NO | | | | # DEFINER | varchar(77) | NO | | | | # SECURITY_TYPE | varchar(7) | NO | | | | # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | # COLLATION_CONNECTION | varchar(32) | NO | | | | my %comments = (); # Retrieve all views my $str = "SELECT TABLE_NAME,VIEW_DEFINITION,CHECK_OPTION,IS_UPDATABLE,DEFINER,SECURITY_TYPE FROM INFORMATION_SCHEMA.VIEWS $condition"; $str .= $self->limit_to_objects('VIEW', 'TABLE_NAME'); $str .= " ORDER BY TABLE_NAME"; $str =~ s/ AND / WHERE /; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %ordered_view = (); my %data = (); while (my $row = $sth->fetch) { $row->[1] =~ s/`$self->{schema}`\.//g; $row->[1] =~ s/ AS `([^`]+\([^`]+)`//ig; $row->[1] =~ s/`([^\s`,]+)`/$1/g; $row->[1] =~ s/"/'/g; $row->[1] =~ s/`/"/g; $data{$row->[0]}{text} = $row->[1]; $data{$row->[0]}{owner} = ''; $data{$row->[0]}{comment} = ''; $data{$row->[0]}{check_option} = $row->[2]; $data{$row->[0]}{updatable} = $row->[3]; $data{$row->[0]}{definer} = $row->[4]; $data{$row->[0]}{security} = $row->[5]; } return %data; } sub _get_triggers { my($self) = @_; # Retrieve all indexes # TRIGGER_CATALOG | varchar(512) | NO | | | | # TRIGGER_SCHEMA | varchar(64) | NO | | | | # TRIGGER_NAME | varchar(64) | NO | | | | # EVENT_MANIPULATION | varchar(6) | NO | | | | # EVENT_OBJECT_CATALOG | varchar(512) | NO | | | | # EVENT_OBJECT_SCHEMA | varchar(64) | NO | | | | # EVENT_OBJECT_TABLE | varchar(64) | NO | | | | # ACTION_ORDER | bigint(4) | NO | | 0 | | # ACTION_CONDITION | longtext | YES | | NULL | | # ACTION_STATEMENT | longtext | NO | | NULL | | # ACTION_ORIENTATION | varchar(9) | NO | | | | # ACTION_TIMING | varchar(6) | NO | | | | # ACTION_REFERENCE_OLD_TABLE | varchar(64) | YES | | NULL | | # ACTION_REFERENCE_NEW_TABLE | varchar(64) | YES | | NULL | | # ACTION_REFERENCE_OLD_ROW | varchar(3) | NO | | | | # ACTION_REFERENCE_NEW_ROW | varchar(3) | NO | | | | # CREATED | datetime | YES | | NULL | | # SQL_MODE | varchar(8192) | NO | | | | # DEFINER | varchar(77) | NO | | | | # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | # COLLATION_CONNECTION | varchar(32) | NO | | | | # DATABASE_COLLATION | varchar(32) | NO | | | | my $str = "SELECT TRIGGER_NAME, ACTION_TIMING, EVENT_MANIPULATION, EVENT_OBJECT_TABLE, ACTION_STATEMENT, '' AS WHEN_CLAUSE, '' AS DESCRIPTION, ACTION_ORIENTATION FROM INFORMATION_SCHEMA.TRIGGERS"; if ($self->{schema}) { $str .= " AND TRIGGER_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','EVENT_OBJECT_TABLE|EVENT_OBJECT_TABLE|TRIGGER_NAME'); $str =~ s/ AND / WHERE /; $str .= " ORDER BY EVENT_OBJECT_TABLE, TRIGGER_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @triggers = (); while (my $row = $sth->fetch) { $row->[7] = 'FOR EACH '. $row->[7]; push(@triggers, [ @$row ]); } return \@triggers; } sub _unique_key { my($self, $table, $owner) = @_; my %result = (); my @accepted_constraint_types = (); push @accepted_constraint_types, "'P'" unless($self->{skip_pkeys}); push @accepted_constraint_types, "'U'" unless($self->{skip_ukeys}); return %result unless(@accepted_constraint_types); # CONSTRAINT_CATALOG | varchar(512) | NO | | | | # CONSTRAINT_SCHEMA | varchar(64) | NO | | | | # CONSTRAINT_NAME | varchar(64) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # CONSTRAINT_TYPE | varchar(64) | NO | | | | my $condition = ''; $condition = " FROM $self->{schema}" if ($self->{schema}); if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/ AND / WHERE /; my %tables_infos = (); if ($table) { $tables_infos{$table} = 1; } else { %tables_infos = Ora2Pg::MySQL::_table_info($self); } # Retrieve all indexes for the given table foreach my $t (keys %tables_infos) { my $sql = "SHOW INDEX FROM `$t` $condition"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $i = 1; while (my $row = $sth->fetch) { # Exclude non unique constraints next if ($row->[1]); #Table : The name of the table. #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. #Seq_in_index : The column sequence number in the index, starting with 1. #Column_name : The column name. #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). #Cardinality : An estimate of the number of unique values in the index. #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. #Packed : Indicates how the key is packed. NULL if it is not. #Null : Contains YES if the column may contain NULL values and '' if not. #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. my $idxname = $row->[0] . '_idx' . $i; if ($row->[2] ne 'PRIMARY') { $idxname = $row->[2]; } my $type = 'P'; $type = 'U' if ($row->[2] ne 'PRIMARY'); next if (!grep(/^'$type'$/, @accepted_constraint_types)); my $generated = 0; $generated = 'GENERATED NAME' if ($row->[2] ne 'PRIMARY'); if (!exists $result{$row->[0]}{$idxname}) { my %constraint = (type => $type, 'generated' => $generated, 'index_name' => $idxname, columns => [ ($row->[4]) ] ); $result{$row->[0]}{$idxname} = \%constraint if ($row->[4]); $i++ if ($row->[2] ne 'PRIMARY'); } else { push(@{$result{$row->[0]}{$idxname}->{columns}}, $row->[4]); } } } return %result; } sub _check_constraint { my ($self, $table, $owner) = @_; if ($self->{db_version} < '8.0.0') { return; } my $condition = ''; $condition .= "AND TABLE_NAME='$table' " if ($table); $condition .= $self->limit_to_objects('CKEY|TABLE', 'CONSTRAINT_NAME|TABLE_NAME'); my $sql = qq{SELECT CONSTRAINT_NAME, TABLE_NAME FROM information_schema.TABLE_CONSTRAINTS WHERE CONSTRAINT_TYPE = 'CHECK' AND TABLE_SCHEMA = '$self->{schema}' $condition}; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { # Pour chaque retour SHOW CREATE TABLE xxxx; my $sql2 = "SHOW CREATE TABLE `$row->[1]`;"; my $sth2 = $self->{dbh}->prepare($sql2) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); # Parsing de CONSTRAINT `CHK_CONSTR` CHECK (((`Age` >= 18) and (`City` = _utf8mb4'Bangalore'))) while (my $r = $sth2->fetch) { $r->[1] =~ s/`//g; my @def = split(/[\r\n]+/, $r->[1]); foreach my $l (@def) { if ($l =~ s/.*CONSTRAINT $row->[0] CHECK (.*)/$1/) { $l =~ s/(LIKE|=) _[^']+('[^']+')/$1 $2/g; $data{$row->[1]}{constraint}{$row->[0]}{condition} = $l; $data{$row->[1]}{constraint}{$row->[0]}{validate} = 'Y'; } } } } return %data; } sub _get_external_tables { my ($self) = @_; # There is no external table in MySQL return; } sub _get_directory { my ($self) = @_; # There is no external table in MySQL return; } sub _get_functions { my $self = shift; # Retrieve all functions # SPECIFIC_NAME | varchar(64) | NO | | | | # ROUTINE_CATALOG | varchar(512) | NO | | | | # ROUTINE_SCHEMA | varchar(64) | NO | | | | # ROUTINE_NAME | varchar(64) | NO | | | | # ROUTINE_TYPE | varchar(9) | NO | | | | # DATA_TYPE | varchar(64) | NO | | | | # or DTD_IDENTIFIER < 5.5 | varchar(64) | NO | | | | # CHARACTER_MAXIMUM_LENGTH | int(21) | YES | | NULL | | # CHARACTER_OCTET_LENGTH | int(21) | YES | | NULL | | # NUMERIC_PRECISION | int(21) | YES | | NULL | | # NUMERIC_SCALE | int(21) | YES | | NULL | | # CHARACTER_SET_NAME | varchar(64) | YES | | NULL | | # COLLATION_NAME | varchar(64) | YES | | NULL | | # DTD_IDENTIFIER | longtext | YES | | NULL | | # ROUTINE_BODY | varchar(8) | NO | | | | # ROUTINE_DEFINITION | longtext | YES | | NULL | | # EXTERNAL_NAME | varchar(64) | YES | | NULL | | # EXTERNAL_LANGUAGE | varchar(64) | YES | | NULL | | # PARAMETER_STYLE | varchar(8) | NO | | | | # IS_DETERMINISTIC | varchar(3) | NO | | | | # SQL_DATA_ACCESS | varchar(64) | NO | | | | # SQL_PATH | varchar(64) | YES | | NULL | | # SECURITY_TYPE | varchar(7) | NO | | | | # CREATED | datetime | NO | | 0000-00-00 00:00:00 | | # LAST_ALTERED | datetime | NO | | 0000-00-00 00:00:00 | | # SQL_MODE | varchar(8192) | NO | | | | # ROUTINE_COMMENT | longtext | NO | | NULL | | # DEFINER | varchar(77) | NO | | | | # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | # COLLATION_CONNECTION | varchar(32) | NO | | | | # DATABASE_COLLATION | varchar(32) | NO | | | | my $str = "SELECT ROUTINE_NAME,ROUTINE_DEFINITION,DATA_TYPE,ROUTINE_BODY,EXTERNAL_LANGUAGE,SECURITY_TYPE,IS_DETERMINISTIC,ROUTINE_TYPE FROM INFORMATION_SCHEMA.ROUTINES"; if ($self->{schema}) { $str .= " AND ROUTINE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('FUNCTION','ROUTINE_NAME'); $str =~ s/ AND / WHERE /; $str .= " ORDER BY ROUTINE_NAME"; # Version below 5.5 do not have DATA_TYPE column it is named DTD_IDENTIFIER if ($self->{db_version} < '5.5.0') { $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; } my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); while (my $row = $sth->fetch) { my $kind = $row->[7]; # FUNCTION or PROCEDURE next if ( ($kind ne $self->{type}) && ($self->{type} ne 'SHOW_REPORT') ); my $sth2 = $self->{dbh}->prepare("SHOW CREATE $kind `$row->[0]`") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { $functions{"$row->[0]"}{text} = $r->[2]; last; } $sth2->finish(); if ($self->{plsql_pgsql} || ($self->{type} eq 'SHOW_REPORT')) { $functions{"$row->[0]"}{name} = $row->[0]; $functions{"$row->[0]"}{return} = $row->[2]; $functions{"$row->[0]"}{definition} = $row->[1]; $functions{"$row->[0]"}{language} = $row->[3]; $functions{"$row->[0]"}{security} = $row->[5]; $functions{"$row->[0]"}{immutable} = $row->[6]; } } return \%functions; } sub _lookup_function { my ($self, $code, $fctname) = @_; my $type = 'functions'; $type = lc($self->{type}) . 's' if ($self->{type} eq 'FUNCTION' or $self->{type} eq 'PROCEDURE'); # Replace all double quote with single quote $code =~ s/"/'/g; # replace backquote with double quote $code =~ s/`/"/g; # Remove some unused code $code =~ s/\s+READS SQL DATA//igs; $code =~ s/\s+UNSIGNED\b((?:.*?)\bFUNCTION\b)/$1/igs; my %fct_detail = (); $fct_detail{func_ret_type} = 'OPAQUE'; # Split data into declarative and code part ($fct_detail{declare}, $fct_detail{code}) = split(/\bBEGIN\b/i, $code, 2); if (!$fct_detail{code}) { if ($fct_detail{declare} =~ s/(RETURN .*)$//i) { $fct_detail{code} = "BEGIN\n $1;\nEND\n"; } } # Remove any label that was before the main BEGIN block $fct_detail{declare} =~ s/\s+[^\s\:]+:\s*$//gs; @{$fct_detail{param_types}} = (); if ( ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\(]+)\s*(\(.*\))\s+RETURNS\s+(.*)//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\(]+)\s*(\(.*\))\s*(.*)//is) ) { $fct_detail{before} = $1; $fct_detail{type} = uc($2); $fct_detail{name} = $3; $fct_detail{args} = $4; my $tmp_returned = $5; $type = lc($fct_detail{type} . 's'); chomp($tmp_returned); if ($tmp_returned =~ s/\b(DECLARE\b.*)//is) { $fct_detail{code} = $1 . $fct_detail{code}; } if ($fct_detail{declare} =~ s/\s*COMMENT\s+(\?TEXTVALUE\d+\?|'[^\']+')//) { $fct_detail{comment} = $1; } elsif ($tmp_returned =~ s/\s*COMMENT\s+(\?TEXTVALUE\d+\?|'[^\']+')//) { $fct_detail{comment} = $1; } $fct_detail{immutable} = 1 if ($fct_detail{declare} =~ s/\s*\bDETERMINISTIC\b//is); $fct_detail{before} = ''; # There is only garbage for the moment $fct_detail{name} =~ s/['"]//g; $fct_detail{fct_name} = $fct_detail{name}; if (!$fct_detail{args}) { $fct_detail{args} = '()'; } $fct_detail{immutable} = 1 if ($fct_detail{return} =~ s/\s*\bDETERMINISTIC\b//is); $fct_detail{immutable} = 1 if ($tmp_returned =~ s/\s*\bDETERMINISTIC\b//is); $fctname = $fct_detail{name} || $fctname; if ($type eq 'functions' && exists $self->{$type}{$fctname}{return} && $self->{$type}{$fctname}{return}) { $fct_detail{hasreturn} = 1; $fct_detail{func_ret_type} = $self->_sql_type($self->{$type}{$fctname}{return}); } elsif ($type eq 'functions' && !exists $self->{$type}{$fctname}{return} && $tmp_returned) { $tmp_returned =~ s/\s+CHARSET.*//is; $fct_detail{func_ret_type} = $self->_sql_type($tmp_returned); $fct_detail{hasreturn} = 1; } $fct_detail{language} = $self->{$type}{$fctname}{language}; $fct_detail{immutable} = 1 if ($self->{$type}{$fctname}{immutable} eq 'YES'); $fct_detail{security} = $self->{$type}{$fctname}{security}; if (!$fct_detail{code} && $tmp_returned) { $tmp_returned .= ';' if ($tmp_returned !~ /;$/s); $fct_detail{code} = "\n$tmp_returned\nEND;" } # Procedure that have out parameters are functions with PG if ($type eq 'procedures' && $fct_detail{args} =~ /\b(OUT|INOUT)\b/) { # set return type to empty to avoid returning void later $fct_detail{func_ret_type} = ' '; } # IN OUT should be INOUT $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; # Move the DECLARE statement from code to the declare section. $fct_detail{declare} = ''; while ($fct_detail{code} =~ s/DECLARE\s+([^;]+;)//is) { $fct_detail{declare} .= "\n$1"; } # Now convert types if ($fct_detail{args}) { $fct_detail{args} = replace_sql_type($self, $fct_detail{args}); } if ($fct_detail{declare}) { $fct_detail{declare} = replace_sql_type($self, $fct_detail{declare}); } $fct_detail{args} =~ s/\s+/ /gs; push(@{$fct_detail{param_types}}, split(/\s*,\s*/, $fct_detail{args})); # Store type used in parameter list to lookup later for custom types map { s/^\(//; } @{$fct_detail{param_types}}; map { s/\)$//; } @{$fct_detail{param_types}}; map { s/\%ORA2PG_COMMENT\d+\%//gs; } @{$fct_detail{param_types}}; map { s/^\s*[^\s]+\s+(IN|OUT|INOUT)/$1/i; s/^((?:IN|OUT|INOUT)\s+[^\s]+)\s+[^\s]*$/$1/i; s/\(.*//; s/\s*\)\s*$//; s/\s+$//; } @{$fct_detail{param_types}}; } else { delete $fct_detail{func_ret_type}; delete $fct_detail{declare}; $fct_detail{code} = $code; } # Mark the function as having out parameters if any my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; my $nbout = $#nout+1 + $#ninout+1; $fct_detail{inout} = 1 if ($nbout > 0); ($fct_detail{code}, $fct_detail{declare}) = replace_mysql_variables($self, $fct_detail{code}, $fct_detail{declare}); # Try to replace LEAVE label by EXIT label my %repl_leave = (); my $i = 0; while ($fct_detail{code} =~ s/LEAVE\s+([^\s;]+)\s*;/%REPEXITLBL$i%/igs) { my $label = $1; if ( $fct_detail{code} =~ /$label:/is) { $repl_leave{$i} = "EXIT $label;"; } else { # This is a main block label $repl_leave{$i} = "RETURN;"; } $fct_detail{code} =~ s/$label:/<<$label>>/igs; $i++; } foreach $i (keys %repl_leave) { $fct_detail{code} =~ s/\%REPEXITLBL$i\%/$repl_leave{$i}/gs; } %repl_leave = (); # Try to replace ITERATE label by CONTINUE label my %repl_iterate = (); $i = 0; while ($fct_detail{code} =~ s/\bITERATE\s+([^\s;]+)\s*;/%REPITERLBL$i%/igs) { my $label = $1; $repl_iterate{$i} = "CONTINUE $label;"; $fct_detail{code} =~ s/$label:/<<$label>>/igs; $i++; } foreach $i (keys %repl_iterate) { $fct_detail{code} =~ s/\%REPITERLBL$i\%/$repl_iterate{$i}/gs; } %repl_iterate = (); # Remove %ROWTYPE from return type $fct_detail{func_ret_type} =~ s/\%ROWTYPE//igs; return %fct_detail; } sub replace_mysql_variables { my ($self, $code, $declare) = @_; # Look for mysql global variables and add them to the custom variable list while ($code =~ s/\b(?:SET\s+)?\@\@(?:SESSION\.)?([^\s:=]+)\s*:=\s*([^;]+);/PERFORM set_config('$1', $2, false);/is) { my $n = $1; my $v = $2; $self->{global_variables}{$n}{name} = lc($n); # Try to set a default type for the variable $self->{global_variables}{$n}{type} = 'bigint'; if ($v =~ /'[^\']*'/) { $self->{global_variables}{$n}{type} = 'varchar'; } if ($n =~ /datetime/i) { $self->{global_variables}{$n}{type} = 'timestamp'; } elsif ($n =~ /time/i) { $self->{global_variables}{$n}{type} = 'time'; } elsif ($n =~ /date/i) { $self->{global_variables}{$n}{type} = 'date'; } } my @to_be_replaced = (); # Look for local variable definition and append them to the declare section while ($code =~ s/SET\s+\@([^\s:]+)\s*:=\s*([^;]+);/SET $1 = $2;/is) { my $n = $1; my $v = $2; # Try to set a default type for the variable my $type = 'integer'; $type = 'varchar' if ($v =~ /'[^']*'/); if ($n =~ /datetime/i) { $type = 'timestamp'; } elsif ($n =~ /time/i) { $type = 'time'; } elsif ($n =~ /date/i) { $type = 'date'; } $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); push(@to_be_replaced, $n); } # Look for local variable definition and append them to the declare section while ($code =~ s/(\s+)\@([^\s:=]+)\s*:=\s*([^;]+);/$1$2 := $3;/is) { my $n = $2; my $v = $3; # Try to set a default type for the variable my $type = 'integer'; $type = 'varchar' if ($v =~ /'[^']*'/); if ($n =~ /datetime/i) { $type = 'timestamp'; } elsif ($n =~ /time/i) { $type = 'time'; } elsif ($n =~ /date/i) { $type = 'date'; } $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); push(@to_be_replaced, $n); } # Fix other call to the same variable in the code foreach my $n (@to_be_replaced) { $code =~ s/\@$n\b(\s*[^:])/$n$1/gs; } # Look for local variable definition and append them to the declare section while ($code =~ s/\@([a-z0-9_]+)/$1/is) { my $n = $1; # Try to set a default type for the variable my $type = 'varchar'; if ($n =~ /datetime/i) { $type = 'timestamp'; } elsif ($n =~ /time/i) { $type = 'time'; } elsif ($n =~ /date/i) { $type = 'date'; } $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); # Fix other call to the same variable in the code $code =~ s/\@$n\b/$n/gs; } # Look for variable definition with SELECT statement $code =~ s/\bSET\s+([^\s=]+)\s*=\s*([^;]+\bSELECT\b[^;]+);/$1 = $2;/igs; # Fix mysql DECLARE clause after BEGIN clause while ($code =~ s/(BEGIN\s+)DECLARE(\s+[^\;]+;)/$1/) { $declare .= "$2\n"; } return ($code, $declare); } sub _list_all_functions { my $self = shift; # Retrieve all functions # ROUTINE_SCHEMA | varchar(64) | NO | | | | # ROUTINE_NAME | varchar(64) | NO | | | | # ROUTINE_TYPE | varchar(9) | NO | | | | my $str = "SELECT ROUTINE_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.ROUTINES"; if ($self->{schema}) { $str .= " AND ROUTINE_SCHEMA = '$self->{schema}'"; } if ($self->{db_version} < '5.5.0') { $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; } $str .= " " . $self->limit_to_objects('FUNCTION','ROUTINE_NAME'); $str =~ s/ AND / WHERE /; $str .= " ORDER BY ROUTINE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @functions = (); while (my $row = $sth->fetch) { push(@functions, $row->[0]); } $sth->finish(); return @functions; } sub _sql_type { my ($self, $type, $len, $precision, $scale, $default, $no_blob_to_oid) = @_; my $data_type = ''; chomp($type); # Simplify timestamp type $type =~ s/TIMESTAMP\s*\(\s*\d+\s*\)/TIMESTAMP/i; $type =~ s/TIME\s*\(\s*\d+\s*\)/TIME/i; $type =~ s/DATE\s*\(\s*\d+\s*\)/DATE/i; # Remove BINARY from CHAR(n) BINARY, TEXT(n) BINARY, VARCHAR(n) BINARY ... $type =~ s/(CHAR|TEXT)\s*(\(\s*\d+\s*\)) BINARY/$1$2/i; $type =~ s/(CHAR|TEXT)\s+BINARY/$1/i; # Some length and scale may have not been extracted before if ($type =~ s/\(\s*(\d+)\s*\)//) { $len = $1; } elsif ($type =~ s/\(\s*(\d+)\s*,\s*(\d+)\s*\)//) { $len = $1; $scale = $2; } if ($type !~ /CHAR/i) { $precision = $len if (!$precision); } # Override the length #$len = $precision if ( ((uc($type) eq 'NUMBER') || (uc($type) eq 'BIT')) && $precision ); $len = $precision if ($precision); if (exists $self->{data_type}{uc($type)}) { $type = uc($type); # Force uppercase $len *= 2 if ($len > 0 && $self->{double_max_varchar} && $type =~ /VARCHAR/); if ($len) { if ( ($type eq "CHAR") || ($type =~ /VARCHAR/) ) { # Type CHAR have default length set to 1 # Type VARCHAR(2) must have a specified length $len = 1 if (!$len && ($type eq "CHAR")); return "$self->{data_type}{$type}($len)"; } elsif ($type eq 'BIT') { if ($precision) { return "$self->{data_type}{$type}($precision)"; } else { return $self->{data_type}{$type}; } } elsif ($type =~ /(TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT|REAL|DOUBLE|FLOAT|DECIMAL|NUMERIC)/i) { if ($type =~ /(TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT)/i) { $scale = 0; $precision = 0; } # This is an integer if (!$scale) { if ($type =~ /UNSIGNED/ && $precision) { # Replace MySQL type UNSIGNED in cast $type =~ s/TINYINT UNSIGNED/smallint/igs; $type =~ s/SMALLINT UNSIGNED/integer/igs; $type =~ s/MEDIUMINT UNSIGNED/integer/igs; $type =~ s/BIGINT UNSIGNED/numeric($precision)/igs; $type =~ s/INT UNSIGNED/bigint/igs; return $type; } elsif ($precision) { if ($type =~ /(REAL|DOUBLE|FLOAT)/i) { return $self->{data_type}{$type}; } elsif ($self->{pg_integer_type}) { if ($precision < 5) { return 'smallint'; } elsif ($precision <= 9) { return 'integer'; # The speediest in PG } else { return 'bigint'; } } return "numeric($precision)"; } else { return $self->{data_type}{$type}; } } else { if ($precision) { if ($type !~ /(DOUBLE|DECIMAL)/ && $self->{pg_numeric_type}) { if ($precision <= 6) { return 'real'; } else { return 'double precision'; } } return "decimal($precision,$scale)"; } } } return $self->{data_type}{$type}; } else { return $self->{data_type}{$type}; } } return $type; } sub replace_sql_type { my ($self, $str) = @_; $str =~ s/with local time zone/with time zone/igs; $str =~ s/([A-Z])ORA2PG_COMMENT/$1 ORA2PG_COMMENT/igs; # Remove any reference to UNSIGNED AND ZEROFILL # but translate CAST( ... AS unsigned) before. $str =~ s/(\s+AS\s+)UNSIGNED/$1$self->{data_type}{'UNSIGNED'}/gis; $str =~ s/\b(UNSIGNED|ZEROFILL)\b//gis; # Remove BINARY from CHAR(n) BINARY and VARCHAR(n) BINARY $str =~ s/(CHAR|TEXT)\s*(\(\s*\d+\s*\))\s+BINARY/$1$2/gis; $str =~ s/(CHAR|TEXT)\s+BINARY/$1/gis; # Replace type with precision my $mysqltype_regex = ''; foreach (keys %{$self->{data_type}}) { $mysqltype_regex .= quotemeta($_) . '|'; } $mysqltype_regex =~ s/\|$//; while ($str =~ /(.*)\b($mysqltype_regex)\s*\(([^\)]+)\)/i) { my $backstr = $1; my $type = uc($2); my $args = $3; if (uc($type) eq 'ENUM') { # Prevent from infinit loop $str =~ s/\(/\%\|/s; $str =~ s/\)/\%\|\%/s; next; } if (exists $self->{data_type}{"$type($args)"}) { $str =~ s/\b$type\($args\)/$self->{data_type}{"$type($args)"}/igs; next; } if ($backstr =~ /_$/) { $str =~ s/\b($mysqltype_regex)\s*\(([^\)]+)\)/$1\%\|$2\%\|\%/is; next; } my ($precision, $scale) = split(/,/, $args); $scale ||= 0; my $len = $precision || 0; $len =~ s/\D//; if ( $type =~ /CHAR/i ) { # Type CHAR have default length set to 1 # Type VARCHAR must have a specified length $len = 1 if (!$len && ($type eq "CHAR")); $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$len\%\|\%/is; } elsif ($precision && ($type =~ /(BIT|TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT|REAL|DOUBLE|FLOAT|DECIMAL|NUMERIC)/)) { if (!$scale) { if ($type =~ /(BIT|TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT)/) { if ($self->{pg_integer_type}) { if ($precision < 5) { $str =~ s/\b$type\b\s*\([^\)]+\)/smallint/is; } elsif ($precision <= 9) { $str =~ s/\b$type\b\s*\([^\)]+\)/integer/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/bigint/is; } } else { $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$precision\%\|\%/i; } } else { $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$precision\%\|\%/is; } } else { if ($type =~ /DOUBLE/) { $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$args\%\|\%/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$args\%\|\%/is; } } } else { # Prevent from infinit loop $str =~ s/\(/\%\|/s; $str =~ s/\)/\%\|\%/s; } } $str =~ s/\%\|\%/\)/gs; $str =~ s/\%\|/\(/gs; # Replace datatype even without precision my %recover_type = (); my $i = 0; foreach my $type (sort { length($b) <=> length($a) } keys %{$self->{data_type}}) { # Keep enum as declared, we are not in table definition next if (uc($type) eq 'ENUM'); while ($str =~ s/\b$type\b/%%RECOVER_TYPE$i%%/is) { $recover_type{$i} = $self->{data_type}{$type}; $i++; } } foreach $i (keys %recover_type) { $str =~ s/\%\%RECOVER_TYPE$i\%\%/$recover_type{$i}/; } # Set varchar without length to text $str =~ s/\bVARCHAR(\s*(?!\())/text$1/igs; return $str; } sub _get_job { my($self) = @_; # Retrieve all database job from user_jobs table my $str = "SELECT EVENT_NAME,EVENT_DEFINITION,EXECUTE_AT FROM INFORMATION_SCHEMA.EVENTS WHERE STATUS = 'ENABLED'"; if ($self->{schema}) { $str .= " AND EVENT_SCHEMA = '$self->{schema}'"; } $str .= $self->limit_to_objects('JOB', 'EVENT_NAME'); $str .= " ORDER BY EVENT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $data{$row->[0]}{what} = $row->[1]; $data{$row->[0]}{interval} = $row->[2]; } return %data; } sub _get_dblink { my($self) = @_; # Must be able to read mysql.servers table return if ($self->{user_grants}); # Retrieve all database link from dba_db_links table my $str = "SELECT OWNER,SERVER_NAME,USERNAME,HOST,DB,PORT,PASSWORD FROM mysql.servers"; $str .= $self->limit_to_objects('DBLINK', 'SERVER_NAME'); $str .= " ORDER BY SERVER_NAME"; $str =~ s/mysql.servers AND /mysql.servers WHERE /; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $data{$row->[1]}{owner} = $row->[0]; $data{$row->[1]}{username} = $row->[2]; $data{$row->[1]}{host} = $row->[3]; $data{$row->[1]}{db} = $row->[4]; $data{$row->[1]}{port} = $row->[5]; $data{$row->[1]}{password} = $row->[6]; } return %data; } =head2 _get_partitions This function implements an MySQL-native partitions information. Return two hash ref with partition details and partition default. =cut sub _get_partitions { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT TABLE_NAME, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_METHOD, PARTITION_EXPRESSION FROM INFORMATION_SCHEMA.PARTITIONS WHERE PARTITION_NAME IS NOT NULL }; $str .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); if ($self->{schema}) { $str .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; } $str .= "ORDER BY TABLE_NAME,PARTITION_ORDINAL_POSITION\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); my %default = (); while (my $row = $sth->fetch) { if ($row->[3] =~ /^MAXVALUE(?:,MAXVALUE)*$/ || $row->[3] eq 'DEFAULT') { $default{$row->[0]}{name} = $row->[2]; next; } $parts{$row->[0]}{$row->[1]}{name} = $row->[2]; $row->[6] =~ s/\`//g; $row->[3] =~ s/\`//g; # Partition by KEY can be converted to HASH partition # but we need to gather the PK or UK definition my @ucol = (); if ($row->[5] =~ /KEY/) { $row->[5] = 'HASH'; if ($row->[6]) { $row->[5] = 'HASH COLUMNS'; } else { my $sql = "SHOW INDEX FROM `$row->[0]`"; my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { if ($r->[2] eq 'PRIMARY') { push(@ucol, $r->[4]) if (!grep(/^$r->[4]$/, @ucol)); } elsif (!$r->[1]) { push(@ucol, $r->[4]) if (!grep(/^$r->[4]$/, @ucol)); } } $sth2->finish; } } else { push(@ucol, $row->[6]); } if ($row->[5] =~ s/ COLUMNS//) { my $i = 0; foreach my $c (split(',', $row->[6])) { push(@{$parts{$row->[0]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $c, 'colpos' => $i, 'tablespace' => $row->[4], 'owner' => ''}); $i++; } } elsif ($row->[5] eq 'HASH') { for (my $i = 0; $i <= $#ucol;$i++) { push(@{$parts{$row->[0]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $ucol[$i], 'colpos' => $i, 'tablespace' => $row->[4], 'owner' => '' }); } } else { @{$parts{$row->[0]}{$row->[1]}{info}} = ( { 'type' => $row->[5], 'value' => $row->[3], 'expression' => $row->[6], 'colpos' => 0, 'tablespace' => $row->[4], 'owner' => '' } ); } } $sth->finish; return \%parts, \%default; } =head2 _get_subpartitions This function implements a MySQL subpartitions information. Return two hash ref with partition details and partition default. =cut sub _get_subpartitions { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT TABLE_NAME, SUBPARTITION_ORDINAL_POSITION, SUBPARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, SUBPARTITION_METHOD, SUBPARTITION_EXPRESSION,PARTITION_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NOT NULL AND SUBPARTITION_EXPRESSION IS NOT NULL }; $str .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); if ($self->{schema}) { $str .= " AND TABLE_SCHEMA ='$self->{schema}'\n"; } $str .= " ORDER BY TABLE_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %subparts = (); my %default = (); while (my $row = $sth->fetch) { if ($row->[3] =~ /^MAXVALUE(?:,MAXVALUE)*$/ || $row->[3] eq 'DEFAULT') { $default{$row->[0]} = $row->[2]; next; } $subparts{$row->[0]}{$row->[7]}{$row->[1]}{name} = $row->[2]; my $i = 0; $row->[6] =~ s/\`//g; $row->[3] =~ s/\`//g; $row->[5] =~ s/ COLUMNS//; $row->[5] = 'HASH' if ($row->[5] =~ /KEY/); foreach my $c (split(',', $row->[6])) { push(@{$subparts{$row->[0]}{$row->[7]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $c, 'colpos' => $i, 'tablespace' => $row->[4], 'owner' => ''}); $i++; } } $sth->finish; return \%subparts, \%default; } =head2 _get_partitions_list This function implements a MySQL-native partitions information. Return a hash of the partition table_name => type =cut sub _get_partitions_list { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT TABLE_NAME, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_METHOD FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NULL AND PARTITION_NAME IS NOT NULL }; $str .= $self->limit_to_objects('TABLE|PARTITION','TABLE_NAME|PARTITION_NAME'); if ($self->{schema}) { $str .= " AND TABLE_SCHEMA ='$self->{schema}'"; } $str .= " ORDER BY TABLE_NAME,PARTITION_NAME\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { $parts{$row->[5]}++; } $sth->finish; return %parts; } =head2 _get_partitioned_table Return a hash of the partitioned table with the number of partition =cut sub _get_partitioned_table { my ($self, %subpart) = @_; # Retrieve all partitions. my $str = qq{ SELECT TABLE_NAME, PARTITION_METHOD, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_EXPRESSION FROM INFORMATION_SCHEMA.PARTITIONS WHERE PARTITION_NAME IS NOT NULL AND (PARTITION_METHOD LIKE 'RANGE%' OR PARTITION_METHOD LIKE 'LIST%' OR PARTITION_METHOD LIKE 'HASH%' OR PARTITION_METHOD LIKE 'KEY%' OR PARTITION_METHOD LIKE 'LINEAR KEY%') }; $str .= $self->limit_to_objects('TABLE|PARTITION','TABLE_NAME|PARTITION_NAME'); if ($self->{schema}) { $str .= " AND TABLE_SCHEMA ='$self->{schema}'"; } $str .= " ORDER BY TABLE_NAME,PARTITION_NAME\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { $parts{"\L$row->[0]\E"}{count} = 0; $parts{"\L$row->[0]\E"}{composite} = 0; if (exists $subpart{"\L$row->[0]\E"}) { $parts{"\L$row->[0]\E"}{composite} = 1; foreach my $k (keys %{$subpart{"\L$row->[0]\E"}}) { $parts{"\L$row->[0]\E"}{count} += $subpart{"\L$row->[0]\E"}{$k}{count}; } $parts{"\L$row->[0]\E"}{count}++; } else { $parts{"\L$row->[0]\E"}{count}++; } $parts{"\L$row->[0]\E"}{type} = $row->[1]; $row->[6] =~ s/\`//g; if ($row->[1] =~ /KEY/) { $parts{"\L$row->[0]\E"}{type} = 'HASH'; if ($row->[6]) { $parts{"\L$row->[0]\E"}{type} = 'HASH COLUMNS'; } else { my $sql = "SHOW INDEX FROM `$row->[0]`"; my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @ucol = (); while (my $r = $sth2->fetch) { #Table : The name of the table. #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. #Seq_in_index : The column sequence number in the index, starting with 1. #Column_name : The column name. #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). #Cardinality : An estimate of the number of unique values in the index. #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. #Packed : Indicates how the key is packed. NULL if it is not. #Null : Contains YES if the column may contain NULL values and '' if not. #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. if ($r->[2] eq 'PRIMARY') { push(@{ $parts{"\L$row->[0]\E"}{columns} }, $r->[4]) if (!grep(/^$r->[4]$/, @{ $parts{"\L$row->[0]\E"}{columns} })); } elsif (!$r->[1]) { push(@ucol, $r->[4]) if (!grep(/^$r->[4]$/, @ucol)); } } $sth2->finish; if ($#{ $parts{"\L$row->[0]\E"}{columns} } < 0) { if ($#ucol >= 0) { push(@{ $parts{"\L$row->[0]\E"}{columns} }, @ucol); } else { $row->[6] =~ s/[\(\)\s]//g; @{ $parts{"\L$row->[0]\E"}{columns} } = split(',', $row->[6]); } } } } if ($parts{"\L$row->[0]\E"}{type} =~ s/ COLUMNS//) { $row->[6] =~ s/[\(\)\s]//g; @{ $parts{"\L$row->[0]\E"}{columns} } = split(',', $row->[6]); } elsif ($row->[6]) { $parts{"\L$row->[0]\E"}{expression} = $row->[6]; } } $sth->finish; return %parts; } =head2 _get_objects This function retrieves all object the Oracle information =cut sub _get_objects { my $self = shift; my %infos = (); # TABLE my $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}'"; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{TABLE}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # VIEW $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.VIEWS WHERE TABLE_SCHEMA = '$self->{schema}'"; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{VIEW}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # TRIGGER $sql = "SELECT TRIGGER_NAME FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_SCHEMA = '$self->{schema}'"; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{TRIGGER}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # INDEX foreach my $t (@{$infos{TABLE}}) { my $sql = "SHOW INDEX FROM `$t->{name}` FROM $self->{schema}"; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my @row = $sth->fetchrow()) { next if ($row[2] eq 'PRIMARY'); push(@{$infos{INDEX}}, { ( name => $row[2], invalid => 0) }); } } # FUNCTION $sql = "SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_TYPE = 'FUNCTION' AND ROUTINE_SCHEMA = '$self->{schema}'"; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{FUNCTION}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # PROCEDURE $sql = "SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_TYPE = 'PROCEDURE' AND ROUTINE_SCHEMA = '$self->{schema}'"; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{PROCEDURE}}, { ( name => $row[0], invalid => 0) }); } $sth->finish(); # PARTITION. my $str = qq{ SELECT TABLE_NAME||'_'||PARTITION_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NULL AND (SUBPARTITION_METHOD LIKE 'RANGE%' OR SUBPARTITION_METHOD LIKE 'LIST%' OR SUBPARTITION_METHOD LIKE 'HASH%' OR SUBPARTITION_METHOD LIKE 'KEY%' OR SUBPARTITION_METHOD LIKE 'LINEAR KEY%') }; $sql .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); if ($self->{schema}) { $sql .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; } $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{'TABLE PARTITION'}}, { ( name => $row[0], invalid => 0) }); } $sth->finish; # SUBPARTITION. $str = qq{ SELECT TABLE_NAME||'_'||SUBPARTITION_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NOT NULL }; $sql .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|SUBPARTITION_NAME'); if ($self->{schema}) { $sql .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; } $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while ( my @row = $sth->fetchrow()) { push(@{$infos{'TABLE PARTITION'}}, { ( name => $row[0], invalid => 0) }); } $sth->finish; return %infos; } sub _get_privilege { my($self) = @_; my %privs = (); my %roles = (); # Retrieve all privilege per table defined in this database my $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,IS_GRANTABLE FROM INFORMATION_SCHEMA.TABLE_PRIVILEGES"; if ($self->{schema}) { $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); $str .= " ORDER BY TABLE_NAME, GRANTEE"; my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved grants\n\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { # Remove the host part of the user $row->[0] =~ s/\@.*//; $row->[0] =~ s/'//g; $privs{$row->[1]}{type} = $row->[2]; if ($row->[3] eq 'YES') { $privs{$row->[1]}{grantable} = $row->[3]; } $privs{$row->[1]}{owner} = ''; push(@{$privs{$row->[1]}{privilege}{$row->[0]}}, $row->[2]); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); # Retrieve all privilege per column table defined in this database $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,COLUMN_NAME,IS_GRANTABLE FROM INFORMATION_SCHEMA.COLUMN_PRIVILEGES"; if ($self->{schema}) { $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $row->[0] =~ s/\@.*//; $row->[0] =~ s/'//g; $privs{$row->[1]}{owner} = ''; push(@{$privs{$row->[1]}{column}{$row->[3]}{$row->[0]}}, $row->[2]); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); return (\%privs, \%roles); } =head2 _get_database_size This function retrieves the size of the MySQL database in MB =cut sub _get_database_size { my $self = shift; my $mb_size = ''; my $condition = ''; my $sql = qq{ SELECT TABLE_SCHEMA "DB Name", sum(DATA_LENGTH + INDEX_LENGTH)/1024/1024 "DB Size in MB" FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='$self->{schema}' GROUP BY TABLE_SCHEMA }; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $mb_size = sprintf("%.2f MB", $row[1]); last; } $sth->finish(); return $mb_size; } =head2 _get_largest_tables This function retrieves the list of largest table of the Oracle database in MB =cut sub _get_largest_tables { my $self = shift; my %table_size = (); my $sql = qq{ SELECT TABLE_NAME, sum(DATA_LENGTH + INDEX_LENGTH)/1024/1024 AS TSize FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='$self->{schema}' }; $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); $sql .= " GROUP BY TABLE_NAME ORDER BY tsize"; $sql .= " LIMIT $self->{top_max}" if ($self->{top_max}); my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute(@{$self->{query_bind_params}}) or return undef; while ( my @row = $sth->fetchrow()) { $table_size{$row[0]} = $row[1]; } $sth->finish(); return %table_size; } sub _get_audit_queries { my($self) = @_; return if (!$self->{audit_user}); my @users = (); push(@users, split(/[,;\s]/, lc($self->{audit_user}))); # Retrieve all object with tablespaces. my $str = "SELECT argument FROM mysql.general_log WHERE command_type='Query' AND argument REGEXP '^(INSERT|UPDATE|DELETE|SELECT)'"; if (($#users >= 0) && !grep(/^all$/, @users)) { $str .= " AND user_host REGEXP '(" . join("'|'", @users) . ")'"; } my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved audited queries\n\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tmp_queries = (); while (my $row = $sth->fetch) { $self->_remove_comments(\$row->[0]); $row->[0] = $self->normalize_query($row->[0]); $tmp_queries{$row->[0]}++; $self->logit(".",1); } $sth->finish; $self->logit("\n", 1); my %queries = (); my $i = 1; foreach my $q (keys %tmp_queries) { $queries{$i} = $q; $i++; } return %queries; } sub _get_synonyms { my ($self) = shift; return; } sub _get_tablespaces { my ($self) = shift; return; } sub _list_tablespaces { my ($self) = shift; return; } sub _get_sequences { my ($self) = shift; return; } sub _extract_sequence_info { my ($self) = shift; return; } # MySQL does not have sequences but we count auto_increment as sequences sub _count_sequences { my $self = shift; # Table: information_schema.tables # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # TABLE_TYPE | varchar(64) | NO | | | | # ENGINE | varchar(64) | YES | | NULL | | # VERSION | bigint(21) unsigned | YES | | NULL | | # ROW_FORMAT | varchar(10) | YES | | NULL | | # TABLE_ROWS | bigint(21) unsigned | YES | | NULL | | # AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | # INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | | # DATA_FREE | bigint(21) unsigned | YES | | NULL | | # AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | | # CREATE_TIME | datetime | YES | | NULL | | # UPDATE_TIME | datetime | YES | | NULL | | # CHECK_TIME | datetime | YES | | NULL | | # TABLE_COLLATION | varchar(32) | YES | | NULL | | # CHECKSUM | bigint(21) unsigned | YES | | NULL | | # CREATE_OPTIONS | varchar(255) | YES | | NULL | | # TABLE_COMMENT | varchar(2048) | NO | | | | my %seqs = (); my $sql = "SELECT TABLE_NAME, AUTO_INCREMENT FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}' AND AUTO_INCREMENT IS NOT NULL"; $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { push(@{$seqs{$row->[0]}}, @$row); } $sth->finish(); return \%seqs; } sub _column_attributes { my ($self, $table, $owner, $objtype) = @_; $objtype ||= 'TABLE'; my $condition = ''; if ($self->{schema}) { $condition .= "AND TABLE_SCHEMA='$self->{schema}' "; } $condition .= "AND TABLE_NAME='$table' " if ($table); if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/^AND/WHERE/; # TABLE_CATALOG | varchar(512) | NO | | | | # TABLE_SCHEMA | varchar(64) | NO | | | | # TABLE_NAME | varchar(64) | NO | | | | # COLUMN_NAME | varchar(64) | NO | | | | # ORDINAL_POSITION | bigint(21) unsigned | NO | | 0 | | # COLUMN_DEFAULT | longtext | YES | | NULL | | # IS_NULLABLE | varchar(3) | NO | | | | # DATA_TYPE | varchar(64) | NO | | | | # CHARACTER_MAXIMUM_LENGTH | bigint(21) unsigned | YES | | NULL | | # CHARACTER_OCTET_LENGTH | bigint(21) unsigned | YES | | NULL | | # NUMERIC_PRECISION | bigint(21) unsigned | YES | | NULL | | # NUMERIC_SCALE | bigint(21) unsigned | YES | | NULL | | # CHARACTER_SET_NAME | varchar(32) | YES | | NULL | | # COLLATION_NAME | varchar(32) | YES | | NULL | | # COLUMN_TYPE | longtext | NO | | NULL | | # COLUMN_KEY | varchar(3) | NO | | | | # EXTRA | varchar(27) | NO | | | | # PRIVILEGES | varchar(80) | NO | | | | # COLUMN_COMMENT | varchar(1024) | NO | | | | my $sql = qq{SELECT COLUMN_NAME, IS_NULLABLE, (CASE WHEN COLUMN_DEFAULT IS NOT NULL THEN COLUMN_DEFAULT ELSE EXTRA END) AS COLUMN_DEFAULT, TABLE_NAME, DATA_TYPE, ORDINAL_POSITION FROM INFORMATION_SCHEMA.COLUMNS $condition ORDER BY ORDINAL_POSITION }; my $sth = $self->{dbh}->prepare($sql); if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $row->[2] =~ s/^_[^']+\\'(.*)\\'/'$1'/; # fix collation on string $data{$row->[3]}{"$row->[0]"}{nullable} = $row->[1]; $data{$row->[3]}{"$row->[0]"}{default} = $row->[2]; # Store the data type of the column following its position $data{$row->[3]}{data_type}{$row->[5]} = $row->[4]; } return %data; } sub _list_triggers { my($self) = @_; my $str = "SELECT TRIGGER_NAME, EVENT_OBJECT_TABLE FROM INFORMATION_SCHEMA.TRIGGERS"; if ($self->{schema}) { $str .= " AND TRIGGER_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','EVENT_OBJECT_TABLE|EVENT_OBJECT_TABLE|TRIGGER_NAME'); $str =~ s/ AND / WHERE /; $str .= " ORDER BY EVENT_OBJECT_TABLE, TRIGGER_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %triggers = (); while (my $row = $sth->fetch) { push(@{$triggers{$row->[1]}}, $row->[0]); } return %triggers; } sub _global_temp_table_info { my($self) = @_; return; } sub _encrypted_columns { my ($self, $table, $owner) = @_; return; } sub _get_subpartitioned_table { my($self) = @_; # Retrieve all partitions. my $str = qq{ SELECT TABLE_NAME, SUBPARTITION_METHOD, SUBPARTITION_ORDINAL_POSITION, PARTITION_NAME, SUBPARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, SUBPARTITION_EXPRESSION FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NOT NULL AND (SUBPARTITION_METHOD LIKE 'RANGE%' OR SUBPARTITION_METHOD LIKE 'LIST%' OR SUBPARTITION_METHOD LIKE 'HASH%' OR SUBPARTITION_METHOD LIKE 'KEY%' OR SUBPARTITION_METHOD LIKE 'LINEAR KEY%') }; $str .= $self->limit_to_objects('TABLE|PARTITION','TABLE_NAME|SUBPARTITION_NAME'); if ($self->{schema}) { $str .= " AND TABLE_SCHEMA ='$self->{schema}'"; } $str .= " ORDER BY TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{count}++; $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{type} = $row->[1]; $row->[7] =~ s/\`//g; if ($parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{type} =~ /KEY/) { $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{type} = 'HASH'; if ($row->[7]) { $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{type} = 'HASH COLUMNS'; } else { my $sql = "SHOW INDEX FROM `$row->[0]`"; my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @ucol = (); while (my $r = $sth2->fetch) { #Table : The name of the table. #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. #Seq_in_index : The column sequence number in the index, starting with 1. #Column_name : The column name. #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). #Cardinality : An estimate of the number of unique values in the index. #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. #Packed : Indicates how the key is packed. NULL if it is not. #Null : Contains YES if the column may contain NULL values and '' if not. #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. if ($r->[2] eq 'PRIMARY') { push(@{ $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{columns} }, $r->[4]) if (!grep(/^$r->[4]$/, @{ $parts{"\L$row->[0]\E"}{columns} })); } elsif (!$row->[1]) { push(@ucol, $r->[4]) if (!grep(/^$r->[4]$/, @ucol)); } } $sth2->finish; if ($#{ $parts{"\L$row->[0]\E"}{columns} } < 0) { push(@{ $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{columns} }, @ucol); } } } if ($parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{type} =~ s/ COLUMNS//) { $row->[7] =~ s/[\(\)\s]//g; @{ $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{columns} } = split(',', $row->[7]); } elsif ($row->[7]) { $parts{"\L$row->[0]\E"}{"\L$row->[3]\E"}{expression} = $row->[7]; } } $sth->finish; return %parts; } # Replace IF("user_status"=0,"username",NULL) # PostgreSQL (CASE WHEN "user_status"=0 THEN "username" ELSE NULL END) sub replace_if { my $str = shift; # First remove all IN (...) before processing my %in_clauses = (); my $j = 0; while ($str =~ s/\b(IN\s*\([^\(\)]+\))/,\%INCLAUSE$j\%/is) { $in_clauses{$j} = $1; $j++; } while ($str =~ s/\bIF\s*\(((?:(?!\)\s*THEN|\s*SELECT\s+|\bIF\s*\().)*)$/\%IF\%$2/is || $str =~ s/\bIF\s*\(([^\(\)]+)\)(\s+AS\s+)/(\%IF\%)$2/is) { my @if_params = (''); my $stop_learning = 0; my $idx = 1; foreach my $c (split(//, $1)) { $idx++ if (!$stop_learning && $c eq '('); $idx-- if (!$stop_learning && $c eq ')'); if ($idx == 0) { # Do not copy last parenthesis in the output string $c = '' if (!$stop_learning); # Inform the loop that we don't want to process any charater anymore $stop_learning = 1; # We have reach the end of the if() parameter # next character must be restored to the final string. $str .= $c; } elsif ($idx > 0) { # We are parsing the if() parameter part, append # the caracter to the right part of the param array. if ($c eq ',' && ($idx - 1) == 0) { # we are switching to a new parameter push(@if_params, ''); } elsif ($c ne "\n") { $if_params[-1] .= $c; } } } my $case_str = 'CASE '; for (my $i = 1; $i <= $#if_params; $i+=2) { $if_params[$i] =~ s/^\s+//gs; $if_params[$i] =~ s/\s+$//gs; if ($i < $#if_params) { if ($if_params[$i] !~ /INCLAUSE/) { $case_str .= "WHEN $if_params[0] THEN $if_params[$i] ELSE $if_params[$i+1] "; } else { $case_str .= "WHEN $if_params[0] $if_params[$i] THEN $if_params[$i+1] "; } } else { $case_str .= " ELSE $if_params[$i] "; } } $case_str .= 'END '; $str =~ s/\%IF\%/$case_str/s; } $str =~ s/\%INCLAUSE(\d+)\%/$in_clauses{$1}/gs; $str =~ s/\s*,\s*IN\s*\(/ IN \(/igs; return $str; } sub _get_plsql_metadata { my $self = shift; my $owner = shift; # Retrieve all functions my $str = "SELECT ROUTINE_NAME,ROUTINE_SCHEMA,ROUTINE_TYPE,ROUTINE_DEFINITION FROM INFORMATION_SCHEMA.ROUTINES"; if ($self->{schema}) { $str .= " WHERE ROUTINE_SCHEMA = '$self->{schema}'"; } $str .= " ORDER BY ROUTINE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); while (my $row = $sth->fetch) { next if (grep(/^$row->[0]$/i, @fct_done)); push(@fct_done, "$row->[0]"); $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{type} = $row->[2]; $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{text} = $row->[3]; my $sth2 = $self->{dbh}->prepare("SHOW CREATE $row->[2] `$row->[0]`") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{text} = $r->[2]; last; } $sth2->finish(); } $sth->finish(); # Look for functions/procedures foreach my $name (sort keys %{$self->{function_metadata}{'unknown'}{'none'}}) { # Retrieve metadata for this function after removing comments $self->_remove_comments(\$self->{function_metadata}{'unknown'}{'none'}{$name}{text}, 1); $self->{comment_values} = (); $self->{function_metadata}{'unknown'}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; my %fct_detail = $self->_lookup_function($self->{function_metadata}{'unknown'}{'none'}{$name}{text}, $name); if (!exists $fct_detail{name}) { delete $self->{function_metadata}{'unknown'}{'none'}{$name}; next; } delete $fct_detail{code}; delete $fct_detail{before}; %{$self->{function_metadata}{'unknown'}{'none'}{$name}{metadata}} = %fct_detail; delete $self->{function_metadata}{'unknown'}{'none'}{$name}{text}; } } sub _get_security_definer { my ($self, $type) = @_; my %security = (); # Retrieve all functions security information my $str = "SELECT ROUTINE_NAME,ROUTINE_SCHEMA,SECURITY_TYPE,DEFINER FROM INFORMATION_SCHEMA.ROUTINES"; if ($self->{schema}) { $str .= " WHERE ROUTINE_SCHEMA = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('FUNCTION|PROCEDURE', 'ROUTINE_NAME|ROUTINE_NAME'); $str .= " ORDER BY ROUTINE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { next if (!$row->[0]); $security{$row->[0]}{security} = $row->[2]; $security{$row->[0]}{owner} = $row->[3]; } $sth->finish(); return (\%security); } =head2 _get_identities This function retrieve information about IDENTITY columns that must be exported as PostgreSQL serial. =cut sub _get_identities { my ($self) = @_; # nothing to do, AUTO_INCREMENT column are converted to serial/bigserial return; } =head2 _get_materialized_views This function implements a mysql-native materialized views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_materialized_views { my($self) = @_; # nothing to do, materialized view are not supported by MySQL. return; } sub _get_materialized_view_names { my($self) = @_; # nothing to do, materialized view are not supported by MySQL. return; } sub _get_package_function_list { my ($self, $owner) = @_; # not package in MySQL return; } sub _get_procedures { my ($self) = @_; # not package in MySQL return _get_functions($self); } sub _get_types { my ($self, $name) = @_; # Not supported return; } sub _col_count { my ($self, $name) = @_; my $sql = qq{SELECT TABLE_SCHEMA, TABLE_NAME, count(*) FROM INFORMATION_SCHEMA.COLUMNS GROUP BY TABLE_SCHEMA, TABLE_NAME }; my $sth = $self->{dbh}->prepare($sql); if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $data{$row->[1]} = $row->[2]; } return %data; } 1; ora2pg-25.0/lib/Ora2Pg/Oracle.pm000066400000000000000000004323661500113072400162440ustar00rootroot00000000000000package Ora2Pg::Oracle; use vars qw($VERSION); use strict; use POSIX qw(locale_h); use Benchmark; use DBI; use Encode; #set locale to LC_NUMERIC C setlocale(LC_NUMERIC,"C"); $VERSION = '25.0'; # Some function might be excluded from export and assessment. our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); # These definitions can be overriden from configuration # file using the DATA_TYPË configuration directive. our %SQL_TYPE = ( # Oracle only has one flexible underlying numeric type, NUMBER. # Without precision and scale it is set to the PG type float8 # to match all needs 'NUMBER' => 'numeric', # CHAR types limit of 2000 bytes with defaults to 1 if no length # is specified. PG char type has max length set to 8104 so it # should match all needs 'CHAR' => 'char', 'NCHAR' => 'char', # VARCHAR types the limit is 2000 bytes in Oracle 7 and 4000 in # Oracle 8. PG varchar type has max length iset to 8104 so it # should match all needs 'VARCHAR' => 'varchar', 'NVARCHAR' => 'varchar', 'VARCHAR2' => 'varchar', 'NVARCHAR2' => 'varchar', 'STRING' => 'varchar', # The DATE data type is used to store the date and time # information. PG type timestamp should match all needs. 'DATE' => 'timestamp(0)', # Type LONG is like VARCHAR2 but with up to 2Gb. PG type text # should match all needs or if you want you could use blob 'LONG' => 'text', # Character data of variable length 'LONG RAW' => 'bytea', # Types LOB and FILE are like LONG but with up to 4Gb. PG type # text should match all needs or if you want you could use blob # (large object) 'CLOB' => 'text', # A large object containing single-byte characters 'NCLOB' => 'text', # A large object containing national character set data 'BLOB' => 'bytea', # Binary large object # The full path to the external file is returned if destination type is text. # If the destination type is bytea the content of the external file is returned. 'BFILE' => 'bytea', # Locator for external large binary file # RAW column with a length of 16 or 32 bytes are usually GUID, convert them to uuid. 'RAW(16)' => 'uuid', 'RAW(32)' => 'uuid', # The RAW type is presented as hexadecimal characters. The # contents are treated as binary data. Limit of 2000 bytes # PG type text should match all needs or if you want you could # use blob (large object) 'RAW' => 'bytea', 'ROWID' => 'oid', 'UROWID' => 'oid', 'FLOAT' => 'double precision', 'DEC' => 'decimal', 'DECIMAL' => 'decimal', 'DOUBLE PRECISION' => 'double precision', 'INT' => 'integer', 'INTEGER' => 'integer', 'BINARY_INTEGER' => 'integer', 'PLS_INTEGER' => 'integer', 'SMALLINT' => 'smallint', 'REAL' => 'real', 'BINARY_FLOAT' => 'numeric', 'BINARY_DOUBLE' => 'numeric', 'TIMESTAMP' => 'timestamp', 'BOOLEAN' => 'boolean', 'INTERVAL' => 'interval', 'XMLTYPE' => 'xml', 'TIMESTAMP WITH TIME ZONE' => 'timestamp with time zone', 'TIMESTAMP WITH LOCAL TIME ZONE' => 'timestamp with time zone', 'SDO_GEOMETRY' => 'geometry', 'ST_GEOMETRY' => 'geometry', ); our %GTYPE = ( 'UNKNOWN_GEOMETRY' => 'GEOMETRY', 'GEOMETRY' => 'GEOMETRY', 'POINT' => 'POINT', 'LINE' => 'LINESTRING', 'CURVE' => 'LINESTRING', 'POLYGON' => 'POLYGON', 'SURFACE' => 'POLYGON', 'COLLECTION' => 'GEOMETRYCOLLECTION', 'MULTIPOINT' => 'MULTIPOINT', 'MULTILINE' => 'MULTILINESTRING', 'MULTICURVE' => 'MULTILINESTRING', 'MULTIPOLYGON' => 'MULTIPOLYGON', 'MULTISURFACE' => 'MULTIPOLYGON', 'SOLID' => 'SOLID', 'MULTISOLID' => 'MULTISOLID' ); our %ORA2PG_SDO_GTYPE = ( '0' => 'GEOMETRY', '1' => 'POINT', '2' => 'LINESTRING', '3' => 'POLYGON', '4' => 'GEOMETRYCOLLECTION', '5' => 'MULTIPOINT', '6' => 'MULTILINESTRING', '7' => 'MULTIPOLYGON', '8' => 'SOLID', '9' => 'MULTISOLID' ); sub _db_connection { my $self = shift; if (!defined $self->{oracle_pwd}) { eval("use Term::ReadKey;") unless $self->{oracle_user} eq '/'; if (!$@) { $self->{oracle_user} = $self->_ask_username('Oracle') unless (defined $self->{oracle_user}); $self->{oracle_pwd} = $self->_ask_password('Oracle') unless ($self->{oracle_user} eq '/'); } } my $ora_session_mode = ($self->{oracle_user} eq "/" || $self->{oracle_user} eq "sys") ? 2 : undef; $self->logit("ORACLE_HOME = $ENV{ORACLE_HOME}\n", 1); $self->logit("NLS_LANG = $ENV{NLS_LANG}\n", 1); $self->logit("NLS_NCHAR = $ENV{NLS_NCHAR}\n", 1); $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$self->{quiet}); my $dbh = DBI->connect($self->{oracle_dsn}, ( $self->{oracle_user} eq "__SEPS__" ? "" : $self->{oracle_user} ), ( $self->{oracle_pwd} eq "__SEPS__" ? "" : $self->{oracle_pwd} ), { ora_envhp => 0, LongReadLen=>$self->{longreadlen}, LongTruncOk=>$self->{longtruncok}, AutoInactiveDestroy => 1, PrintError => 0, ora_session_mode => $ora_session_mode, ora_client_info => 'ora2pg ' || $VERSION } ); # Check for connection failure if (!$dbh) { $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); } # Get Oracle version, needed to set date/time format my $sth = $dbh->prepare( "SELECT BANNER FROM v\$version" ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $self->{db_version} = $row[0]; last; } $sth->finish(); chomp($self->{db_version}); $self->{db_version} =~ s/ \- .*//; # Check if the connection user has the DBA privilege $sth = $dbh->prepare( "SELECT 1 FROM DBA_ROLE_PRIVS" ); if (!$sth) { my $ret = $dbh->err; if ($ret == 942 && $self->{prefix} eq 'DBA') { $self->logit("HINT: you should activate USER_GRANTS for a connection without DBA privilege. Continuing with USER privilege.\n"); # No DBA privilege, set use of ALL_* tables instead of DBA_* tables $self->{prefix} = 'ALL'; $self->{user_grants} = 1; } } else { $sth->finish(); } # Fix a problem when exporting type LONG and LOB $dbh->{'LongReadLen'} = $self->{longreadlen}; $dbh->{'LongTruncOk'} = $self->{longtruncok}; # Embedded object (user defined type) must be returned as an # array rather than an instance. This is normally the default. $dbh->{'ora_objects'} = 0; # Force datetime format $self->_datetime_format($dbh); # Force numeric format $self->_numeric_format($dbh); # Use consistent reads for concurrent dumping... $dbh->begin_work || $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); if ($self->{debug} && !$self->{quiet}) { $self->logit("Isolation level: $self->{transaction}\n", 1); } $sth = $dbh->prepare($self->{transaction}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->finish; # Get the current SCN to get data if required if (grep(/^$self->{type}$/i, 'INSERT', 'COPY', 'TEST_DATA') && lc($self->{oracle_scn}) eq 'current') { $sth = $dbh->prepare("SELECT CURRENT_SCN FROM v\$database") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my @row = $sth->fetchrow(); $self->{oracle_scn} = $row[0]; $sth->finish; } $self->logit("Using SCN: $self->{oracle_scn}\n", 1) if ($self->{oracle_scn}); # Force execution of initial command $self->_ora_initial_command($dbh); return $dbh; } sub _get_version { my $self = shift; my $oraver = ''; my $sql = "SELECT BANNER FROM v\$version"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $oraver = $row[0]; last; } $sth->finish(); chomp($oraver); $oraver =~ s/ \- .*//; return $oraver; } sub _schema_list { my $self = shift; my $sql = "SELECT DISTINCT OWNER FROM $self->{prefix}_TABLES WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') ORDER BY OWNER"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; $sth; } sub _table_exists { my ($self, $schema, $table) = @_; my $ret = ''; my $sql = "SELECT TABLE_NAME FROM $self->{prefix}_TABLES WHERE OWNER = '$schema' AND TABLE_NAME = '$table'"; my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $ret = $row[0]; } $sth->finish(); return $ret; } =head2 _get_encoding This function retrieves the Oracle database encoding Returns a handle to a DB query statement. =cut sub _get_encoding { my ($self, $dbh) = @_; my $sql = "SELECT * FROM NLS_DATABASE_PARAMETERS"; my $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my $language = ''; my $territory = ''; my $charset = ''; my $nls_timestamp_format = ''; my $nls_date_format = ''; while ( my @row = $sth->fetchrow()) { if ($row[0] eq 'NLS_LANGUAGE') { $language = $row[1]; } elsif ($row[0] eq 'NLS_TERRITORY') { $territory = $row[1]; } elsif ($row[0] eq 'NLS_CHARACTERSET') { $charset = $row[1]; } elsif ($row[0] eq 'NLS_TIMESTAMP_FORMAT') { $nls_timestamp_format = $row[1]; } elsif ($row[0] eq 'NLS_DATE_FORMAT') { $nls_date_format = $row[1]; } } $sth->finish(); $sql = "SELECT * FROM NLS_SESSION_PARAMETERS"; $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); my $ora_encoding = ''; while ( my @row = $sth->fetchrow()) { #$self->logit("SESSION PARAMETERS: $row[0] $row[1]\n", 1); if ($row[0] eq 'NLS_LANGUAGE') { $language = $row[1]; } elsif ($row[0] eq 'NLS_TERRITORY') { $territory = $row[1]; } elsif ($row[0] eq 'NLS_TIMESTAMP_FORMAT') { $nls_timestamp_format = $row[1]; } elsif ($row[0] eq 'NLS_DATE_FORMAT') { $nls_date_format = $row[1]; } } $sth->finish(); $ora_encoding = $language . '_' . $territory . '.' . $charset; my $pg_encoding = auto_set_encoding($charset); return ($ora_encoding, $charset, $pg_encoding, $nls_timestamp_format, $nls_date_format); } # Return the lower value between two sub min { return $_[0] if ($_[0] < $_[1]); return $_[1]; } =head2 _table_info This function retrieves all tables information. Returns a handle to a DB query statement. =cut sub _table_info { my $self = shift; my $do_real_row_count = shift; my $owner = ''; if ($self->{schema}) { $owner .= " A.OWNER='$self->{schema}' "; } else { $owner .= " A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } #### # Get name of all TABLE objects in ALL_OBJECTS looking at OBJECT_TYPE='TABLE' #### my $sql = "SELECT A.OWNER,A.OBJECT_NAME,A.OBJECT_TYPE FROM $self->{prefix}_OBJECTS A WHERE A.OBJECT_TYPE IN ('TABLE','VIEW') AND $owner"; $sql .= $self->limit_to_objects('TABLE', 'A.OBJECT_NAME'); $self->logit("DEBUG: $sql\n", 2); my $t0 = Benchmark->new; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $nrows = 0; my %tbtype = (); while (my $row = $sth->fetch) { $self->{all_objects}{"$row->[0].$row->[1]"} = $row->[2]; $nrows++; } $sth->finish(); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Collecting $nrows tables in $self->{prefix}_OBJECTS took: " . timestr($td) . "\n", 1); #### # Get comments for all tables #### my %comments = (); if ($self->{type} eq 'TABLE') { $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM $self->{prefix}_TAB_COMMENTS A WHERE $owner"; if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); } $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); $self->logit("DEBUG: $sql\n", 2); $t0 = Benchmark->new; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $nrows = 0; my %tbtype = (); while (my $row = $sth->fetch) { next if (!exists $self->{all_objects}{"$row->[3].$row->[0]"} || $self->{all_objects}{"$row->[3].$row->[0]"} ne 'TABLE'); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[3].$row->[0]"; } $comments{$row->[0]}{comment} = $row->[1]; $comments{$row->[0]}{table_type} = $row->[2]; $tbtype{$row->[2]}++; $nrows++; } $sth->finish(); $t1 = Benchmark->new; $td = timediff($t1, $t0); $self->logit("Collecting $nrows tables comments in $self->{prefix}_TAB_COMMENTS took: " . timestr($td) . "\n", 1); } #### # Get information about all tables #### $sql = "SELECT A.OWNER,A.TABLE_NAME,NVL(num_rows,1) NUMBER_ROWS,A.TABLESPACE_NAME,A.NESTED,A.LOGGING,A.PARTITIONED,A.PCT_FREE,A.TEMPORARY,A.DURATION FROM $self->{prefix}_TABLES A WHERE $owner"; $sql .= " AND A.TEMPORARY='N'" if (!$self->{export_gtt} or $self->{type} =~ /^(COPY|INSERT)$/); $sql .= " AND (A.NESTED != 'YES' OR A.LOGGING != 'YES') AND A.SECONDARY = 'N'"; if ($self->{db_version} !~ /Release [89]/) { $sql .= " AND (A.DROPPED IS NULL OR A.DROPPED = 'NO')"; } if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); } $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); $sql .= " AND (A.IOT_TYPE IS NULL OR A.IOT_TYPE = 'IOT')"; #$sql .= " ORDER BY A.OWNER, A.TABLE_NAME"; $self->logit("DEBUG: $sql\n", 2); $t0 = Benchmark->new; $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tables_infos = (); $nrows = 0; while (my $row = $sth->fetch) { next if (!exists $self->{all_objects}{"$row->[0].$row->[1]"} || $self->{all_objects}{"$row->[0].$row->[1]"} ne 'TABLE'); if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } $tables_infos{$row->[1]}{owner} = $row->[0] || ''; $tables_infos{$row->[1]}{num_rows} = $row->[2] || 0; $tables_infos{$row->[1]}{tablespace} = $row->[3] || 0; $tables_infos{$row->[1]}{comment} = $comments{$row->[1]}{comment} || ''; $tables_infos{$row->[1]}{type} = $comments{$row->[1]}{table_type} || ''; $tables_infos{$row->[1]}{nested} = $row->[4] || ''; if ($row->[5] eq 'NO') { $tables_infos{$row->[1]}{nologging} = 1; } else { $tables_infos{$row->[1]}{nologging} = 0; } if ($row->[6] eq 'NO') { $tables_infos{$row->[1]}{partitioned} = 0; } else { $tables_infos{$row->[1]}{partitioned} = 1; } # Only take care of PCTFREE upper than the Oracle default value if (($row->[7] || 0) > 10) { $tables_infos{$row->[1]}{fillfactor} = 100 - min(90, $row->[7]); } # Global temporary table ? $tables_infos{$row->[1]}{temporary} = $row->[8]; $tables_infos{$row->[1]}{duration} = $row->[9]; $nrows++; } $sth->finish(); $t1 = Benchmark->new; $td = timediff($t1, $t0); $self->logit("Collecting $nrows tables information in $self->{prefix}_TABLES took: " . timestr($td) . "\n", 1); return %tables_infos; } sub _column_comments { my ($self, $table) = @_; my $condition = ''; my $sql = "SELECT A.COLUMN_NAME,A.COMMENTS,A.TABLE_NAME,A.OWNER FROM $self->{prefix}_COL_COMMENTS A $condition"; if ($self->{schema}) { $sql .= "WHERE A.OWNER='$self->{schema}' "; } else { $sql .= " WHERE A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $sql .= "AND A.TABLE_NAME='$table' " if ($table); if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); } if (!$table) { $sql .= $self->limit_to_objects('TABLE','TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[2] = "$row->[3].$row->[2]"; } next if (!$self->is_in_struct($row->[2], $row->[0])); $data{$row->[2]}{$row->[0]} = $row->[1]; } return %data; } sub _column_info { my ($self, $table, $owner, $objtype, $recurs, @expanded_views) = @_; $objtype ||= 'TABLE'; my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND A.OWNER='$owner' "; } else { $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } my $sth = ''; my $sql = ''; if ($self->{db_version} !~ /Release 8/) { my $exclude_mview = $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); $sql = qq{ SELECT A.COLUMN_NAME, A.DATA_TYPE, A.DATA_LENGTH, A.NULLABLE, A.DATA_DEFAULT, A.DATA_PRECISION, A.DATA_SCALE, A.CHAR_LENGTH, A.TABLE_NAME, A.OWNER FROM $self->{prefix}_TAB_COLUMNS A WHERE A.TABLE_NAME NOT LIKE 'BIN\$%' $condition ORDER BY A.COLUMN_ID }; $sth = $self->{dbh}->prepare($sql); if (!$sth) { my $ret = $self->{dbh}->err; if (!$recurs && ($ret == 942) && ($self->{prefix} eq 'DBA')) { $self->logit("HINT: Please activate USER_GRANTS or connect using a user with DBA privilege.\n"); $self->{prefix} = 'ALL'; return $self->_column_info($table, $owner, $objtype, 1, @expanded_views); } $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); } } else { # an 8i database. $sql = qq{ SELECT A.COLUMN_NAME, A.DATA_TYPE, A.DATA_LENGTH, A.NULLABLE, A.DATA_DEFAULT, A.DATA_PRECISION, A.DATA_SCALE, A.DATA_LENGTH, A.TABLE_NAME, A.OWNER FROM $self->{prefix}_TAB_COLUMNS A WHERE A.TABLE_NAME NOT LIKE 'BIN\$%' $condition ORDER BY A.COLUMN_ID }; $sth = $self->{dbh}->prepare($sql); if (!$sth) { my $ret = $self->{dbh}->err; if (!$recurs && ($ret == 942) && ($self->{prefix} eq 'DBA')) { $self->logit("HINT: Please activate USER_GRANTS or connect using a user with DBA privilege.\n"); $self->{prefix} = 'ALL'; return $self->_column_info($table, $owner, $objtype, 1, @expanded_views); } $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); } } $self->logit("DEBUG, $sql", 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); # Default number of line to scan to grab the geometry type of the column. # If it not limited, the query will scan the entire table which may take a very long time. my $max_lines = 2; $max_lines = $self->{autodetect_spatial_type} if ($self->{autodetect_spatial_type} > 1); my $spatial_gtype = 'SELECT DISTINCT c.%s.SDO_GTYPE FROM %s c WHERE ROWNUM < ' . $max_lines; my $st_spatial_gtype = "SELECT DISTINCT $self->{st_geometrytype_function}(c.\%s) FROM \%s c WHERE ROWNUM < " . $max_lines; # Set query to retrieve the SRID my $spatial_srid = "SELECT SRID FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME=? AND COLUMN_NAME=? AND OWNER=?"; my $st_spatial_srid = "SELECT $self->{st_srid_function}(c.\%s) FROM \%s c WHERE ROWNUM < 2"; if ($self->{convert_srid}) { # Translate SRID to standard EPSG SRID, may return 0 because there's lot of Oracle only SRID. $spatial_srid = "SELECT sdo_cs.map_oracle_srid_to_epsg(SRID) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME=? AND COLUMN_NAME=? AND OWNER=?"; } # Get the dimension of the geometry by looking at the number of element in the SDO_DIM_ARRAY my $spatial_dim = "SELECT t.SDO_DIMNAME, t.SDO_LB, t.SDO_UB FROM ALL_SDO_GEOM_METADATA m, TABLE (m.diminfo) t WHERE m.TABLE_NAME=? AND m.COLUMN_NAME=? AND OWNER=?"; my $st_spatial_dim = "SELECT $self->{st_dimension_function}(c.\%s) FROM \%s c WHERE ROWNUM < 2"; my $is_virtual_col = "SELECT V.VIRTUAL_COLUMN FROM $self->{prefix}_TAB_COLS V WHERE V.OWNER=? AND V.TABLE_NAME=? AND V.COLUMN_NAME=?"; my $sth3 = undef; if ($self->{db_version} !~ /Release 8/) { $sth3 = $self->{dbh}->prepare($is_virtual_col); } my $t0 = Benchmark->new; my %data = (); my $pos = 0; my $ncols = 0; while (my $row = $sth->fetch) { my $tmptable = $row->[8]; $tmptable = "$row->[9].$row->[8]" if (!$self->{schema} && $self->{export_schema}); # Skip object if it is not in the object list and if this is not # a view or materialized view that must be exported as table. next if (!exists $self->{all_objects}{"$row->[9].$row->[8]"} || ($self->{all_objects}{"$row->[9].$row->[8]"} =~ /^(VIEW|MATERIALIZED VIEW)$/) && !grep(/^$row->[8]$/i, @expanded_views) ); $row->[2] = $row->[7] if $row->[1] =~ /char/i; # Seems that for a NUMBER with a DATA_SCALE to 0, no DATA_PRECISION and a DATA_LENGTH of 22 # Oracle use a NUMBER(38) instead if ( ($row->[1] eq 'NUMBER') && ($row->[6] eq '0') && ($row->[5] eq '') && ($row->[2] == 22) ) { $row->[2] = 38; } # In case we have a default value, check if this is a virtual column my $virtual = 'NO'; if ($self->{pg_supports_virtualcol} and defined $sth3 and $row->[4]) { $sth3->execute($row->[9],$row->[8],$row->[0]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $r = $sth3->fetch; $virtual = $r->[0]; } # check if this is a spatial column (srid, dim, gtype) my @geom_inf = (); if ($row->[1] eq 'SDO_GEOMETRY' || $row->[1] =~ /^ST_|STGEOM_/) { # Get the SRID of the column if ($self->{convert_srid} > 1) { push(@geom_inf, $self->{convert_srid}); } else { my @result = (); if ($row->[1] =~ /^ST_|STGEOM_/) { $spatial_srid = sprintf($st_spatial_srid, $row->[0], "$row->[9].$row->[8]"); } my $sth2 = $self->{dbh}->prepare($spatial_srid); if (!$sth2) { if ($self->{dbh}->errstr !~ /ORA-01741/) { $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); } else { # No SRID defined, use default one $self->logit("WARNING: Error retreiving SRID, no matter default SRID will be used: $spatial_srid\n", 0); } } else { if ($row->[1] =~ /^ST_|STGEOM_/) { $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } else { $sth2->execute($row->[8],$row->[0],$row->[9]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } while (my $r = $sth2->fetch) { push(@result, $r->[0]) if ($r->[0] =~ /\d+/); } $sth2->finish(); } if ($#result == 0) { push(@geom_inf, $result[0]); } elsif ($self->{default_srid}) { push(@geom_inf, $self->{default_srid}); } else { push(@geom_inf, 0); } } # Grab constraint type and dimensions from index definition my $found_contraint = 0; my $found_dims = 0; foreach my $idx (keys %{$self->{tables}{$tmptable}{idx_type}}) { if (exists $self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}) { foreach my $c (@{$self->{tables}{$tmptable}{indexes}{$idx}}) { if ($c eq $row->[0]) { if ($self->{tables}{$tmptable}{idx_type}{$idx}{type_dims}) { $found_dims = $self->{tables}{$tmptable}{idx_type}{$idx}{type_dims}; } if ($self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}) { $found_contraint = $GTYPE{$self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}} || $self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}; } } } } } # Get the dimension of the geometry column if (!$found_dims) { if ($row->[1] =~ /^ST_|STGEOM_/) { $spatial_dim = sprintf($st_spatial_dim, $row->[0], "$row->[9].$row->[8]"); } my $sth2 = $self->{dbh}->prepare($spatial_dim); if (!$sth2) { $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); } if ($row->[1] =~ /^ST_|STGEOM_/) { $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } else { $sth2->execute($row->[8],$row->[0],$row->[9]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } my $count = 0; while (my $r = $sth2->fetch) { $count++; } $sth2->finish(); push(@geom_inf, $count); } else { push(@geom_inf, $found_dims); } # Set dimension and type of the spatial column if (!$found_contraint && $self->{autodetect_spatial_type}) { # Get spatial information my $squery = sprintf($spatial_gtype, $row->[0], "$row->[9].$row->[8]"); if ($row->[1] =~ /^ST_|STGEOM_/) { $squery = sprintf($st_spatial_gtype, $row->[0], "$row->[9].$row->[8]"); } my $sth2 = $self->{dbh}->prepare($squery); if (!$sth2) { $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); } $sth2->execute or $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); my @result = (); while (my $r = $sth2->fetch) { if ($r->[0] =~ /(\d)$/) { push(@result, $Ora2Pg::Oracle::ORA2PG_SDO_GTYPE{$1}); } elsif ($r->[0] =~ /ST_(.*)$/) { push(@result, $1); } } $sth2->finish(); if ($#result == 0) { push(@geom_inf, $result[0]); } else { push(@geom_inf, join(',', @result)); } } elsif ($found_contraint) { push(@geom_inf, $found_contraint); } else { push(@geom_inf, $Ora2Pg::Oracle::ORA2PG_SDO_GTYPE{0}); } } # Replace dot in column name by underscore if ($row->[0] =~ /\./ && (!exists $self->{replaced_cols}{"\L$tmptable\E"} || !exists $self->{replaced_cols}{"\L$tmptable\E"}{"\L$row->[0]\E"})) { $self->{replaced_cols}{"\L$tmptable\E"}{"\L$row->[0]\E"} = $row->[0]; $self->{replaced_cols}{"\L$tmptable\E"}{"\L$row->[0]\E"} =~ s/\./_/g; } if (!$self->{schema} && $self->{export_schema}) { next if (!$self->is_in_struct($tmptable, $row->[0])); push(@{$data{$tmptable}{"$row->[0]"}}, (@$row, $virtual, $pos, @geom_inf)); } else { next if (!$self->is_in_struct($row->[8], $row->[0])); push(@{$data{"$row->[8]"}{"$row->[0]"}}, (@$row, $virtual, $pos, @geom_inf)); } $pos++; $ncols++; } my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Collecting $ncols columns in $self->{prefix}_INDEXES took: " . timestr($td) . "\n", 1); $sth3->finish() if (defined $sth3); return %data; } =head2 _get_fts_indexes_info This function retrieve FTS index attributes informations Returns a hash of containing all useful attribute values for all FTS indexes =cut sub _get_fts_indexes_info { my ($self, $owner) = @_; my $condition = ''; $condition .= "AND IXV_INDEX_OWNER='$owner' " if ($owner); $condition .= $self->limit_to_objects('INDEX', "IXV_INDEX_NAME"); # Retrieve all indexes informations my $sth = $self->{dbh}->prepare(<logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); SELECT DISTINCT IXV_INDEX_OWNER,IXV_INDEX_NAME,IXV_CLASS,IXV_ATTRIBUTE,IXV_VALUE FROM CTXSYS.CTX_INDEX_VALUES WHERE (IXV_CLASS='WORDLIST' AND IXV_ATTRIBUTE='STEMMER') $condition END $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %indexes_info = (); while (my $row = $sth->fetch) { my $save_idx = $row->[1]; if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } $indexes_info{$row->[1]}{"\L$row->[3]\E"} = $row->[4]; } return %indexes_info; } sub _get_indexes { my ($self, $table, $owner, $generated_indexes) = @_; # Retrieve FTS indexes information before. my %idx_info = (); %idx_info = _get_fts_indexes_info($self, $owner) if ($self->_table_exists('CTXSYS', 'CTX_INDEX_VALUES')); my $sub_owner = ''; if ($owner) { $sub_owner = "AND A.INDEX_OWNER=B.TABLE_OWNER"; } my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND A.INDEX_OWNER='$owner' "; } else { $condition .= " AND A.INDEX_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$self->{export_gtt} or $self->{type} =~ /^(COPY|INSERT)$/) { $condition .= " AND B.TEMPORARY = 'N' "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE|INDEX', "A.TABLE_NAME|A.INDEX_NAME"); } else { @{$self->{query_bind_params}} = (); } # When comparing number of index we need to retrieve generated index (mostly PK) my $generated = ''; #$generated = " B.GENERATED = 'N'" if (!$generated_indexes); # Filter on name instead of the GENERATED column because we are # missing the ones that have been created automatically by the # Automatic Indexing feature of Oracle 19c. See issue: #1589 #$generated = " (B.INDEX_NAME NOT LIKE 'SYS\$_C\$_%' escape '\$')" if (!$generated_indexes); my $t0 = Benchmark->new; my $sth = ''; my $sql = ''; if ($self->{db_version} !~ /Release 8/) { my $no_mview = $self->exclude_mviews('A.INDEX_OWNER, A.TABLE_NAME'); $no_mview = '' if ($self->{type} eq 'MVIEW'); $sql = qq{SELECT DISTINCT A.INDEX_NAME,A.COLUMN_NAME,B.UNIQUENESS,A.COLUMN_POSITION,B.INDEX_TYPE,B.TABLE_TYPE,'N' AS GENERATED,B.JOIN_INDEX,A.TABLE_NAME,A.INDEX_OWNER,B.TABLESPACE_NAME,B.ITYP_NAME,B.PARAMETERS,A.DESCEND FROM $self->{prefix}_IND_COLUMNS A JOIN $self->{prefix}_INDEXES B ON (B.INDEX_NAME=A.INDEX_NAME AND B.OWNER=A.INDEX_OWNER) WHERE$generated $condition $no_mview ORDER BY A.COLUMN_POSITION}; } else { # an 8i database. $sql = qq{SELECT DISTINCT A.INDEX_NAME,A.COLUMN_NAME,B.UNIQUENESS,A.COLUMN_POSITION,B.INDEX_TYPE,B.TABLE_TYPE,'N' AS GENERATED, 'NO' AS JOIN_INDEX, A.TABLE_NAME,A.INDEX_OWNER,B.TABLESPACE_NAME,B.ITYP_NAME,B.PARAMETERS,A.DESCEND FROM $self->{prefix}_IND_COLUMNS A, $self->{prefix}_INDEXES B WHERE $generated $condition AND B.INDEX_NAME=A.INDEX_NAME AND B.OWNER=A.INDEX_OWNER ORDER BY A.COLUMN_POSITION}; } $sql =~ s/WHERE\s+AND/WHERE/s; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $idxnc = qq{SELECT IE.COLUMN_EXPRESSION FROM $self->{prefix}_IND_EXPRESSIONS IE, $self->{prefix}_IND_COLUMNS IC WHERE IE.INDEX_OWNER = IC.INDEX_OWNER AND IE.INDEX_NAME = IC.INDEX_NAME AND IE.TABLE_OWNER = IC.TABLE_OWNER AND IE.TABLE_NAME = IC.TABLE_NAME AND IE.COLUMN_POSITION = IC.COLUMN_POSITION AND IC.COLUMN_NAME = ? AND IE.TABLE_NAME = ? AND IC.TABLE_OWNER = ? }; my $sth2 = $self->{dbh}->prepare($idxnc) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); my %unique = (); my %idx_type = (); my %index_tablespace = (); my $nidx = 0; while (my $row = $sth->fetch) { # Exclude log indexes of materialized views, there must be a better # way to exclude then than looking at index name, fill free to fix it. next if ($row->[0] =~ /^I_SNAP\$_/); # Handle case where indexes name include the schema at create time $row->[0] =~ s/^$self->{schema}\.//i if ($self->{schema}); my $save_tb = $row->[8]; if (!$self->{schema} && $self->{export_schema}) { $row->[8] = "$row->[9].$row->[8]"; } next if (!$self->is_in_struct($row->[8], $row->[1])); # Show a warning when an index has the same name as the table if ( !$self->{indexes_renaming} && !$self->{indexes_suffix} && (lc($row->[0]) eq lc($table)) ) { print STDERR "WARNING: index $row->[0] has the same name as the table itself. Please rename it before export or enable INDEXES_RENAMING.\n"; } $unique{$row->[8]}{$row->[0]} = $row->[2]; # Mark the index as autogenerated for constraints if ($row->[1] =~ /^SYS_C_/) { $row->[6] = 'Y'; } # Save original column name my $colname = $row->[1]; # Replace function based index type if ( ($row->[4] =~ /FUNCTION-BASED/i) && ($colname =~ /^SYS_NC\d+\$$/) ) { $sth2->execute($colname,$save_tb,$row->[-5]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $nc = $sth2->fetch(); $row->[1] = $nc->[0]; $row->[1] =~ s/"//g; $row->[1] =~ s/'//g if ($row->[1] =~ /^'[^'\s]+'$/); # Single row constraint based on a constant and a function based unique index if ($nc->[0] =~ /^\d+$/ && $row->[4] =~ /FUNCTION-BASED/i) { $row->[1] = '(' . $nc->[0] . ')'; } # Enclose with double quote if required when is is not an index function elsif ($row->[1] !~ /\(.*\)/ && $row->[4] !~ /FUNCTION-BASED/i) { $row->[1] = $self->quote_object_name($row->[1]); } # Append DESC sort order when not default to ASC if ($row->[13] eq 'DESC') { $row->[1] .= " DESC"; } } else { # Quote column with unsupported symbols $row->[1] = $self->quote_object_name($row->[1]); } $row->[1] =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/$1/isg; # Index with DESC are declared as FUNCTION-BASED, fix that if (($row->[4] =~ /FUNCTION-BASED/i) && ($row->[1] !~ /\(.*\)/)) { $row->[4] =~ s/FUNCTION-BASED\s*//; } $idx_type{$row->[8]}{$row->[0]}{type_name} = $row->[11]; if (($#{$row} > 6) && ($row->[7] eq 'Y')) { $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4] . ' JOIN'; } else { $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4]; } my $idx_name = $row->[0]; if (!$self->{schema} && $self->{export_schema}) { $idx_name = "$row->[9].$row->[0]"; } if (exists $idx_info{$idx_name}) { $idx_type{$row->[8]}{$row->[0]}{stemmer} = $idx_info{$idx_name}{stemmer}; } if ($row->[11] =~ /SPATIAL_INDEX/) { $idx_type{$row->[8]}{$row->[0]}{type} = 'SPATIAL INDEX'; if ($row->[12] =~ /layer_gtype=([^\s,]+)/i) { $idx_type{$row->[8]}{$row->[0]}{type_constraint} = uc($1); } if ($row->[12] =~ /sdo_indx_dims=(\d+)/i) { $idx_type{$row->[8]}{$row->[0]}{type_dims} = $1; } } if ($row->[4] eq 'BITMAP') { $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4]; } push(@{$data{$row->[8]}{$row->[0]}}, $row->[1]); $index_tablespace{$row->[8]}{$row->[0]} = $row->[10]; $nidx++; } $sth->finish(); $sth2->finish(); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Collecting $nidx indexes in $self->{prefix}_INDEXES took: " . timestr($td) . "\n", 1); return \%unique, \%data, \%idx_type, \%index_tablespace; } sub _foreign_key { my ($self, $table, $owner) = @_; my @tmpparams = (); my $condition = ''; $condition .= "AND CONS.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND CONS.OWNER = '$owner' "; } else { $condition .= "AND CONS.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } $condition .= $self->limit_to_objects('FKEY|TABLE','CONS.CONSTRAINT_NAME|CONS.TABLE_NAME'); my $deferrable = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "DEFERRABLE"; my $defer = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "CONS.DEFERRABLE"; my $sql = <{prefix}_CONSTRAINTS CONS LEFT JOIN $self->{prefix}_CONS_COLUMNS COLS ON (COLS.CONSTRAINT_NAME = CONS.CONSTRAINT_NAME AND COLS.OWNER = CONS.OWNER AND COLS.TABLE_NAME = CONS.TABLE_NAME) LEFT JOIN $self->{prefix}_CONSTRAINTS CONS_R ON (CONS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND CONS_R.OWNER = CONS.R_OWNER) LEFT JOIN $self->{prefix}_CONS_COLUMNS COLS_R ON (COLS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND COLS_R.POSITION=COLS.POSITION AND COLS_R.OWNER = CONS.R_OWNER) WHERE CONS.CONSTRAINT_TYPE = 'R' $condition END if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('CONS.OWNER, CONS.TABLE_NAME'); } $sql .= "\nORDER BY CONS.TABLE_NAME, CONS.CONSTRAINT_NAME, COLS.POSITION"; if ($self->{db_version} =~ /Release 8/) { $sql = <{prefix}_CONSTRAINTS CONS, $self->{prefix}_CONS_COLUMNS COLS, $self->{prefix}_CONSTRAINTS CONS_R, $self->{prefix}_CONS_COLUMNS COLS_R WHERE CONS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND CONS_R.OWNER = CONS.R_OWNER AND COLS.CONSTRAINT_NAME = CONS.CONSTRAINT_NAME AND COLS.OWNER = CONS.OWNER AND COLS.TABLE_NAME = CONS.TABLE_NAME AND COLS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND COLS_R.POSITION=COLS.POSITION AND COLS_R.OWNER = CONS.R_OWNER AND CONS.CONSTRAINT_TYPE = 'R' $condition ORDER BY CONS.TABLE_NAME, CONS.CONSTRAINT_NAME, COLS.POSITION END } my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); my %data = (); my %link = (); #my @tab_done = (); my $i = 0; while (my $row = $sth->fetch) { my $local_table = $row->[0]; my $remote_table = $row->[3]; if (!$self->{schema} && $self->{export_schema}) { $local_table = "$row->[10].$row->[0]"; $remote_table = "$row->[11].$row->[3]"; } next if (!$self->is_in_struct($local_table, $row->[2])); next if (!$self->is_in_struct($remote_table, $row->[2])); #$row->[1] = $row->[1] . $i++ if (exists $link{$local_table}{$row->[1]); push(@{$data{$local_table}}, [ ($row->[1],$row->[4],$row->[6],$row->[7],$row->[8],$row->[9],$row->[11],$row->[0],$row->[10],$row->[14]) ]); # TABLENAME CONSTNAME COLNAME push(@{$link{$local_table}{$row->[1]}{local}}, $row->[2]); # TABLENAME CONSTNAME TABLENAME COLNAME push(@{$link{$local_table}{$row->[1]}{remote}{$remote_table}}, $row->[5]); } return \%link, \%data; } sub _get_ref_key_info { my ($self, $constname, $owner) = @_; my $sql = <{prefix}_CONS_COLUMNS A JOIN $self->{prefix}_CONSTRAINTS C ON A.CONSTRAINT_NAME = C.CONSTRAINT_NAME JOIN $self->{prefix}_tab_columns b ON a.owner = b.owner AND a.table_name = b.table_name AND a.column_name = b.column_name LEFT JOIN $self->{prefix}_constraints cons_r on (cons_r.constraint_name = c.r_constraint_name and cons_r.owner = c.r_owner) LEFT JOIN $self->{prefix}_part_tables p on (cons_r.table_name = p.table_name and cons_r.owner = p.owner) WHERE A.CONSTRAINT_NAME='$constname' AND A.OWNER = '$owner' END my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute() or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { $data{refconstraint} = $row->[4]; $data{reftable} = $row->[0]; $data{refcolumn} = $row->[1]; $data{reftype} = $row->[2]; $data{reflengh} = $row->[3]; $data{refprecision} = $row->[5]; $data{refscale} = $row->[6]; $data{refcharlengh} = $row->[7]; $data{refrtable} = $row->[8]; $data{refparttype} = $row->[9]; } return %data; } =head2 _alias_info This function implements an Oracle-native column information. Returns a list of array references containing the following information for each alias of the specified view: [( column name, column id )] =cut sub _alias_info { my ($self, $view) = @_; my $str = "SELECT COLUMN_NAME, COLUMN_ID, OWNER FROM $self->{prefix}_TAB_COLUMNS WHERE TABLE_NAME='$view'"; if ($self->{schema}) { $str .= " AND OWNER = '$self->{schema}'"; } else { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $str .= " ORDER BY COLUMN_ID ASC"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $data = $sth->fetchall_arrayref(); #$self->logit("View $view column aliases:\n", 1); foreach my $d (@$data) { if (!$self->{schema} && $self->{export_schema}) { $d->[0] = "$d->[2].$d->[0]"; } #$self->logit("\t$d->[0] => column id:$d->[1]\n", 1); } return @$data; } =head2 _get_views This function implements an Oracle-native views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_views { my ($self) = @_; my $owner = ''; if ($self->{schema}) { $owner = "AND A.OWNER='$self->{schema}' "; } else { $owner = "AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } #### # Get name of all VIEW objects in ALL_OBJECTS looking at OBJECT_TYPE='VIEW' or OBJECT_TYPE='MVIEW' #### my $sql = "SELECT A.OWNER,A.OBJECT_NAME,A.OBJECT_TYPE FROM $self->{prefix}_OBJECTS A WHERE A.OBJECT_TYPE IN ('VIEW', 'MATERIALIZED VIEW') $owner"; if (!$self->{export_invalid}) { $sql .= " AND A.STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $sql .= " AND A.STATUS <> 'VALID'"; } $sql .= $self->limit_to_objects('VIEW', 'A.OBJECT_NAME'); $self->logit("DEBUG: $sql\n", 2); my $t0 = Benchmark->new; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my $nrows = 0; my %tbtype = (); my %all_objects = (); while (my $row = $sth->fetch) { $all_objects{"$row->[0].$row->[1]"} = $row->[2]; $nrows++; } $sth->finish(); my $t1 = Benchmark->new; my $td = timediff($t1, $t0); $self->logit("Collecting $nrows tables in $self->{prefix}_OBJECTS took: " . timestr($td) . "\n", 1); my %comments = (); if ($self->{type} ne 'SHOW_REPORT') { $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM $self->{prefix}_TAB_COMMENTS A WHERE 1=1 $owner"; $sql .= $self->limit_to_objects('VIEW', 'A.TABLE_NAME'); $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { next if ($row->[2] ne 'VIEW'); next if (scalar keys %{ $self->{all_objects} } > 0 && !exists $self->{all_objects}{"$row->[3].$row->[0]"}); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[3].$row->[0]"; } $comments{$row->[0]}{comment} = $row->[1]; $comments{$row->[0]}{table_type} = $row->[2]; } $sth->finish(); } # Retrieve all views my $str = "SELECT v.VIEW_NAME,v.TEXT,v.OWNER FROM $self->{prefix}_VIEWS v"; if (!$self->{schema}) { $str .= " WHERE v.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE v.OWNER = '$self->{schema}'"; } $str .= $self->limit_to_objects('VIEW', 'v.VIEW_NAME'); # Compute view order, where depended view appear before using view my %view_order = (); if ($self->{type} ne 'SHOW_REPORT' && !$self->{no_view_ordering}) { if ($self->{db_version} !~ /Release (8|9|10|11\.1)/) { if ($self->{schema}) { $owner = "AND o.OWNER='$self->{schema}' "; } else { $owner = "AND o.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } $sql = qq{ WITH x (ITER, OWNER, OBJECT_NAME) AS ( SELECT 1 , o.OWNER, o.OBJECT_NAME FROM $self->{prefix}_OBJECTS o WHERE OBJECT_TYPE = 'VIEW' $owner AND NOT EXISTS (SELECT 1 FROM $self->{prefix}_DEPENDENCIES d WHERE TYPE LIKE 'VIEW' AND REFERENCED_TYPE = 'VIEW' AND REFERENCED_OWNER = o.OWNER AND d.OWNER = o.OWNER and o.OBJECT_NAME=d.NAME) UNION ALL SELECT ITER + 1, d.OWNER, d.NAME FROM $self->{prefix}_DEPENDENCIES d JOIN x ON d.REFERENCED_OWNER = x.OWNER and d.REFERENCED_NAME = x.OBJECT_NAME WHERE TYPE LIKE 'VIEW' AND REFERENCED_TYPE = 'VIEW' ) SELECT max(ITER) ITER, OWNER, OBJECT_NAME FROM x GROUP BY OWNER, OBJECT_NAME ORDER BY ITER ASC, 2, 3 }; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $view_order{"\U$row->[1].$row->[2]\E"} = $row->[0]; } $sth->finish(); } } $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { next if (!exists $all_objects{"$row->[2].$row->[0]"}); if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[2].$row->[0]"; } $data{$row->[0]}{text} = $row->[1]; $data{$row->[0]}{owner} = $row->[2]; $data{$row->[0]}{comment} = $comments{$row->[0]}{comment} || ''; if ($self->{type} ne 'SHOW_REPORT') { @{$data{$row->[0]}{alias}} = _alias_info ($self, $row->[0]); } if ($self->{type} ne 'SHOW_REPORT' && exists $view_order{"\U$row->[2].$row->[0]\E"}) { $data{$row->[0]}{iter} = $view_order{"\U$row->[2].$row->[0]\E"}; } } return %data; } sub _get_triggers { my($self) = @_; # Retrieve all triggers my $str = "SELECT T.TRIGGER_NAME, T.TRIGGER_TYPE, T.TRIGGERING_EVENT, T.TABLE_NAME, T.TRIGGER_BODY, T.WHEN_CLAUSE, T.DESCRIPTION, T.ACTION_TYPE, T.OWNER, T.STATUS FROM $self->{prefix}_TRIGGERS T JOIN $self->{prefix}_OBJECTS O ON (T.TRIGGER_NAME = O.OBJECT_NAME AND T.OWNER = O.OWNER) WHERE O.OBJECT_TYPE = 'TRIGGER'"; if (!$self->{export_invalid}) { $str .= " AND O.STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND O.STATUS <> 'VALID'"; } if (!$self->{schema}) { $str .= " AND T.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND T.OWNER = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','T.TABLE_NAME|T.TABLE_NAME|T.TRIGGER_NAME'); #$str .= " ORDER BY TABLE_NAME, TRIGGER_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @triggers = (); while (my $row = $sth->fetch) { push(@triggers, [ @$row ]); } return \@triggers; } sub _unique_key { my ($self, $table, $owner, $type) = @_; my %result = (); my @accepted_constraint_types = (); if ($type) { push @accepted_constraint_types, "'$type'"; } else { push @accepted_constraint_types, "'P'" unless($self->{skip_pkeys}); push @accepted_constraint_types, "'U'" unless($self->{skip_ukeys}); } return %result unless(@accepted_constraint_types); my $cons_types = '('. join(',', @accepted_constraint_types) .')'; my $indexname = "'' AS INDEX_NAME"; if ($self->{db_version} !~ /Release 8/) { $indexname = 'B.INDEX_NAME'; } # Get columns of all the table in the specified schema or excluding the list of system schema my $sql = qq{SELECT DISTINCT A.COLUMN_NAME,A.CONSTRAINT_NAME,A.OWNER,A.POSITION,B.CONSTRAINT_NAME,B.CONSTRAINT_TYPE,B.DEFERRABLE,B.DEFERRED,B.GENERATED,B.TABLE_NAME,B.OWNER,$indexname FROM $self->{prefix}_CONS_COLUMNS A JOIN $self->{prefix}_CONSTRAINTS B ON (B.CONSTRAINT_NAME = A.CONSTRAINT_NAME AND B.OWNER = A.OWNER) }; if ($owner) { $sql .= " WHERE A.OWNER = '$owner'"; } else { $sql .= " WHERE A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $sql .= " AND B.CONSTRAINT_TYPE IN $cons_types"; $sql .= " AND B.TABLE_NAME='$table'" if ($table); $sql .= " AND B.STATUS='ENABLED' "; if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('B.OWNER, B.TABLE_NAME'); } # Get the list of constraints in the specified schema or excluding the list of system schema my @tmpparams = (); if ($self->{type} ne 'SHOW_REPORT') { $sql .= $self->limit_to_objects('UKEY|TABLE', 'B.CONSTRAINT_NAME|B.TABLE_NAME'); push(@tmpparams, @{$self->{query_bind_params}}) if (defined $self->{query_bind_params}); $sql .= $self->limit_to_objects('UKEY', 'B.CONSTRAINT_NAME'); push(@tmpparams, @{$self->{query_bind_params}}) if (defined $self->{query_bind_params}); } $sql .= " ORDER BY A.POSITION"; my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@tmpparams) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { my $name = $row->[9]; if (!$self->{schema} && $self->{export_schema}) { $name = "$row->[10].$row->[9]"; } if (!exists $result{$name}{$row->[4]}) { $result{$name}{$row->[4]} = { (type => $row->[5], 'generated' => $row->[8], 'index_name' => $row->[11], 'deferrable' => $row->[6], 'deferred' => $row->[7], columns => ()) }; push(@{ $result{$name}{$row->[4]}->{columns} }, $row->[0]) if ($row->[4] !~ /^SYS_NC/i); } elsif ($row->[4] !~ /^SYS_NC/i) { push(@{ $result{$name}{$row->[4]}->{columns} }, $row->[0]); } } return %result; } sub _check_constraint { my ($self, $table, $owner) = @_; my $condition = ''; $condition .= "AND TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND OWNER = '$owner' "; } else { $condition .= "AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } $condition .= $self->limit_to_objects('CKEY|TABLE', 'CONSTRAINT_NAME|TABLE_NAME'); my $sql = qq{ SELECT A.CONSTRAINT_NAME,A.R_CONSTRAINT_NAME,A.SEARCH_CONDITION,A.DELETE_RULE,A.DEFERRABLE,A.DEFERRED,A.R_OWNER,A.TABLE_NAME,A.OWNER,A.VALIDATED FROM $self->{prefix}_CONSTRAINTS A WHERE A.CONSTRAINT_TYPE='C' $condition AND A.STATUS='ENABLED' }; if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); } my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $row->[7] = "$row->[8].$row->[7]"; } $data{$row->[7]}{constraint}{$row->[0]}{condition} = $row->[2]; $data{$row->[7]}{constraint}{$row->[0]}{validate} = $row->[9]; } return %data; } sub _get_external_tables { my ($self) = @_; # Retrieve all database link from dba_db_links table my $str = "SELECT a.*,b.DIRECTORY_PATH,c.LOCATION,a.OWNER FROM $self->{prefix}_EXTERNAL_TABLES a, $self->{prefix}_DIRECTORIES b, $self->{prefix}_EXTERNAL_LOCATIONS c"; if (!$self->{schema}) { $str .= " WHERE a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE a.OWNER = '$self->{schema}'"; } $str .= " AND a.DEFAULT_DIRECTORY_NAME = b.DIRECTORY_NAME AND a.TABLE_NAME=c.TABLE_NAME AND a.DEFAULT_DIRECTORY_NAME=c.DIRECTORY_NAME AND a.OWNER=c.OWNER"; $str .= $self->limit_to_objects('TABLE', 'a.TABLE_NAME'); #$str .= " ORDER BY a.TABLE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } $data{$row->[1]}{directory} = $row->[5]; $data{$row->[1]}{directory_path} = $row->[10]; if ($data{$row->[1]}{directory_path} =~ /([\/\\])/) { $data{$row->[1]}{directory_path} .= $1 if ($data{$row->[1]}{directory_path} !~ /$1$/); } $data{$row->[1]}{location} = $row->[11]; $data{$row->[1]}{delimiter} = ','; if ($row->[8] =~ /FIELDS TERMINATED BY '(.)'/is) { $data{$row->[1]}{delimiter} = $1; } if ($row->[8] =~ /PREPROCESSOR EXECDIR\s*:\s*'([^']+)'/is) { $data{$row->[1]}{program} = $1; } } $sth->finish(); return %data; } sub _get_directory { my ($self) = @_; # Retrieve all database link from dba_db_links table my $str = "SELECT d.DIRECTORY_NAME, d.DIRECTORY_PATH, d.OWNER, p.GRANTEE, p.PRIVILEGE FROM $self->{prefix}_DIRECTORIES d, $self->{prefix}_TAB_PRIVS p"; $str .= " WHERE d.DIRECTORY_NAME = p.TABLE_NAME"; if (!$self->{schema}) { $str .= " AND p.GRANTEE NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND p.GRANTEE = '$self->{schema}'"; } $str .= $self->limit_to_objects('TABLE', 'd.DIRECTORY_NAME'); #$str .= " ORDER BY d.DIRECTORY_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[2].$row->[0]"; } $data{$row->[0]}{path} = $row->[1]; if ($row->[1] !~ /\/$/) { $data{$row->[0]}{path} .= '/'; } $data{$row->[0]}{grantee}{$row->[3]} .= $row->[4]; } $sth->finish(); return %data; } sub _get_functions { my $self = shift; # Retrieve all functions my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='FUNCTION'"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if (!$self->{schema}) { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND OWNER = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('FUNCTION','OBJECT_NAME'); #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } next if (grep(/^$row->[0]$/i, @fct_done)); push(@fct_done, $row->[0]); $functions{"$row->[0]"}{owner} = $row->[1]; } $sth->finish(); my $sql = "SELECT NAME,OWNER,TEXT FROM $self->{prefix}_SOURCE"; if (!$self->{schema}) { $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " WHERE OWNER = '$self->{schema}'"; } $sql .= " " . $self->limit_to_objects('FUNCTION','NAME'); $sql .= " ORDER BY OWNER,NAME,LINE"; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } # Fix possible Malformed UTF-8 character $row->[2] = encode('UTF-8', $row->[2]) if ($self->{force_plsql_encoding}); # Remove some bargage when migrating from 8i $row->[2] =~ s/\bAUTHID\s+[^\s]+\s+//is; if (exists $functions{"$row->[0]"}) { $functions{"$row->[0]"}{text} .= $row->[2]; } } return \%functions; } sub _lookup_function { my ($self, $plsql, $pname, $meta) = @_; my %fct_detail = (); $fct_detail{func_ret_type} = 'OPAQUE'; # Split data into declarative and code part ($fct_detail{declare}, $fct_detail{code}) = split(/\bBEGIN\b/i, $plsql, 2); return if (!$fct_detail{code}); @{$fct_detail{param_types}} = (); $fct_detail{declare} =~ s/(\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; $fct_detail{declare} =~ s/RETURN\%ORA2PG_COMMENT\d+\%/RETURN/is; if ( ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s*(\([^\)]*\))//is) || ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s+(RETURN|IS|AS)/$4/is) ) { $fct_detail{before} = $1; $fct_detail{type} = uc($2); $fct_detail{name} = $3; $fct_detail{args} = $4; $fct_detail{fct_name} = $3; $fct_detail{fct_name} =~ s/^[^\.]+\.//; $fct_detail{fct_name} =~ s/"//g; # When the function comes from a package remove global declaration # outside comments. They have already been extracted before. if ($pname && $fct_detail{before}) { $self->_remove_comments(\$fct_detail{before}); my $cmt = ''; while ($fct_detail{before} =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { # only keep comment $cmt .= $1; } $fct_detail{before} = $cmt; } if ($fct_detail{args} =~ /\b(RETURN|IS|AS)\b/is) { $fct_detail{args} = '()'; } my $clause = ''; my $code = ''; $fct_detail{name} =~ s/"//g; $fct_detail{immutable} = 1 if ($fct_detail{declare} =~ s/\bDETERMINISTIC\b//is); $fct_detail{setof} = 1 if ($fct_detail{declare} =~ s/\bPIPELINED\b//is); $fct_detail{declare} =~ s/\bDEFAULT\b/:=/igs; if ($fct_detail{declare} =~ s/(.*?)\bRETURN\s+self\s+AS RESULT IS//is) { $fct_detail{args} .= $1; $fct_detail{hasreturn} = 1; $fct_detail{func_ret_type} = 'OPAQUE'; } elsif ($fct_detail{declare} =~ s/(.*?)\bRETURN\s+([^\s]+)//is) { $fct_detail{args} .= $1; $fct_detail{hasreturn} = 1; my $ret_typ = $2 || ''; $ret_typ =~ s/(\%ORA2PG_COMMENT\d+\%)+//i; $fct_detail{orig_func_ret_type} = $ret_typ; $fct_detail{func_ret_type} = $self->_sql_type($ret_typ) || 'OPAQUE'; } if ($fct_detail{declare} =~ s/(.*?)(USING|AS|IS)(\s+(?!REF\s+))/$3/is) { $fct_detail{args} .= $1 if (!$fct_detail{hasreturn}); $clause = $2; } $fct_detail{args} =~ s/;.*//s; $fct_detail{declare} =~ s/\s*AS\%ORA2PG_COMMENT\d+\%//is; if ($fct_detail{declare} =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { $fct_detail{language} = $1; if ($fct_detail{declare} =~ /LIBRARY\s+([^\s="'><\!\(\)]+)/is) { $fct_detail{library} = $1; } if ($fct_detail{declare} =~ /NAME\s+"([^"]+)"/is) { $fct_detail{library_fct} = $1; } } #### # rewrite argument syntax #### # Replace alternate syntax for default value $fct_detail{args} =~ s/:=/DEFAULT/igs; my $json_arg = $fct_detail{args}; $json_arg =~ s/^\s*\(\s*//s; $json_arg =~ s/\s*\)\s*$//s; my %params_rewrite = (); my $y = 0; while ($json_arg =~ s/(\([^\(\)]+\))/%ARGPARAM$y%/s) { $params_rewrite{$y} = $1; $y++; } my $routine_name = $fct_detail{name}; if ($self->{type} eq 'PACKAGE' && $pname) { $routine_name = $pname . '.' . $fct_detail{name}; } $self->{json_config} = qq/ { "routine_type": "\U$fct_detail{type}\E", "ora": { "routine_name": "\U$routine_name\E", "return_type": "$fct_detail{orig_func_ret_type}", "args_list": [ { "0": [ /; my @json_args = split(/\s*,\s*/, $json_arg); map { s/\%ARGPARAM(\d+)\%/$params_rewrite{$1}/sg; } @json_args; foreach my $a (@json_args) { my $arg_name = ''; my $arg_kind = 'IN'; my $arg_default = ''; if ($a =~ s/^([^\s]+)\s+//s) { $arg_name = $1; } if ($a =~ s/^(IN\s+OUT|OUT|IN)\s+//s) { $arg_kind = $1; } if ($a =~ s/\s*DEFAULT\s+(.*)//s) { $arg_default = $1; } my $arg_type = $a; $self->{json_config} .= qq/ { "name": "$arg_name", "mode": "$arg_kind", "type": "$arg_type", "default": "$arg_default", "value": "" }, /; } $self->{json_config} =~ s/,$//s; $self->{json_config} .= qq/ ] } ] }, /; # NOCOPY not supported $fct_detail{args} =~ s/\s*NOCOPY//igs; # IN OUT should be INOUT $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; # Remove %ROWTYPE from arguments, we can use the table name as type $fct_detail{args} =~ s/\%ROWTYPE//igs; # Replace DEFAULT EMPTY_BLOB() from function/procedure arguments by DEFAULT NULL $fct_detail{args} =~ s/\s+DEFAULT\s+EMPTY_[CB]LOB\(\)/DEFAULT NULL/igs; # Now convert types $fct_detail{args} = Ora2Pg::PLSQL::replace_sql_type($self, $fct_detail{args}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, $self->{varchar_to_text}, %{$self->{data_type}}); $fct_detail{declare} = Ora2Pg::PLSQL::replace_sql_type($self, $fct_detail{declare}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, $self->{varchar_to_text}, %{$self->{data_type}}); $json_arg = $fct_detail{args}; $json_arg =~ s/^\s*\(\s*//s; $json_arg =~ s/\s*\)\s*$//s; %params_rewrite = (); $y = 0; while ($json_arg =~ s/(\([^\(\)]+\))/%ARGPARAM$y%/s) { $params_rewrite{$y} = $1; $y++; } $self->{json_config} .= qq/ "pg": { "routine_name": "\L$routine_name\E", "return_type": "$fct_detail{func_ret_type}", "args_list": [ { "0": [ /; @json_args = split(/\s*,\s*/, $json_arg); map { s/\%ARGPARAM(\d+)\%/$params_rewrite{$1}/sg; } @json_args; foreach my $a (@json_args) { my $arg_name = ''; my $arg_kind = 'IN'; my $arg_default = ''; if ($a =~ s/^([^\s]+)\s+//s) { $arg_name = $1; } if ($a =~ s/^(IN\s+OUT|OUT|IN)\s+//s) { $arg_kind = $1; } if ($a =~ s/\s*DEFAULT\s+(.*)//s) { $arg_default = $1; } my $arg_type = $a; $self->{json_config} .= qq/ { "name": "$arg_name", "mode": "$arg_kind", "type": "$arg_type", "default": "$arg_default", "value": "" }, /; } $self->{json_config} =~ s/,$//s; $self->{json_config} .= qq/ ] } ] } }, /; # Sometime variable used in FOR ... IN SELECT loop is not declared # Append its RECORD declaration in the DECLARE section. my $tmp_code = $fct_detail{code}; while ($tmp_code =~ s/\bFOR\s+([^\s]+)\s+IN(.*?)LOOP//is) { my $varname = quotemeta($1); my $clause = $2; if ($fct_detail{declare} !~ /\b$varname\s+/is) { chomp($fct_detail{declare}); # When the cursor is refereing to a statement, declare # it as record otherwise it don't need to be replaced if ($clause =~ /\bSELECT\b/is) { $fct_detail{declare} .= "\n $varname RECORD;\n"; } } } # Set parameters for AUTONOMOUS TRANSACTION $fct_detail{args} =~ s/\s+/ /gs; push(@{$fct_detail{at_args}}, split(/\s*,\s*/, $fct_detail{args})); # Remove type parts to only get parameter's name push(@{$fct_detail{param_types}}, @{$fct_detail{at_args}}); map { s/\s(IN|OUT|INOUT)\s/ /i; } @{$fct_detail{at_args}}; map { s/^\(//; } @{$fct_detail{at_args}}; map { s/^\s+//; } @{$fct_detail{at_args}}; map { s/\s.*//; } @{$fct_detail{at_args}}; map { s/\)$//; } @{$fct_detail{at_args}}; @{$fct_detail{at_args}} = grep(/^.+$/, @{$fct_detail{at_args}}); # Store type used in parameter list to lookup later for custom types map { s/^\(//; } @{$fct_detail{param_types}}; map { s/\)$//; } @{$fct_detail{param_types}}; map { s/\%ORA2PG_COMMENT\d+\%//gs; } @{$fct_detail{param_types}}; map { s/^\s*[^\s]+\s+(IN|OUT|INOUT)/$1/i; s/^((?:IN|OUT|INOUT)\s+[^\s]+)\s+[^\s]*$/$1/i; s/\(.*//; s/\s*\)\s*$//; s/\s+$//; } @{$fct_detail{param_types}}; } else { delete $fct_detail{func_ret_type}; delete $fct_detail{declare}; $fct_detail{code} = $plsql; } # PostgreSQL procedure do not support OUT parameter, translate them into INOUT params if (!$fct_detail{hasreturn} && $self->{pg_supports_procedure} && !$self->{pg_supports_outparam} && ($fct_detail{args} =~ /\bOUT\s+[^,\)]+/i)) { $fct_detail{args} =~ s/\bOUT(\s+[^,\)]+)/INOUT$1/igs; } # Mark the function as having out parameters if any my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; my $nbout = $#nout+1 + $#ninout+1; $fct_detail{inout} = 1 if ($nbout > 0); # Mark function as having custom type in parameter list if ($fct_detail{inout} and $nbout > 1) { foreach my $t (@{$fct_detail{param_types}}) { # Consider column type reference to never be a composite type this # is clearly not right but the false positive case might be very low next if ($t =~ /\%TYPE/i || ($t !~ s/^(OUT|INOUT)\s+//i)); # Mark out parameter as using composite type if (!grep(/^\Q$t\E$/i, 'int', 'bigint', 'date', values %SQL_TYPE, values %ORA2PG_SDO_GTYPE)) { $fct_detail{inout}++; } } } # Collect user defined function while ($fct_detail{declare} =~ s/\b([^\s]+)\s+EXCEPTION\s*;//) { my $e = lc($1); if (!exists $self->{custom_exception}{$e}) { $self->{custom_exception}{$e} = $self->{exception_id}++; } } $fct_detail{declare} =~ s/PRAGMA\s+EXCEPTION_INIT[^;]*;//igs; # Replace call to global variables declared in this package foreach my $n (keys %{$self->{global_variables}}) { next if (!$n || ($pname && (uc($n) !~ /^\U$pname\E\./))); my $tmpname = $n; $tmpname =~ s/^$pname\.//i; next if ($fct_detail{code} !~ /\b\Q$tmpname\E\b/is); my $i = 0; while ($fct_detail{code} =~ s/(SELECT\s+(?:.*?)\s+)INTO\s+\Q$tmpname\E\s+([^;]+);/PERFORM set_config('$n', ($1$2), false);/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/\b$n\s*:=\s*([^;]+)\s*;/PERFORM set_config('$n', $1, false);/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^\.]+)\b$self->{global_variables}{$n}{name}\s*:=\s*([^;]+);/$1PERFORM set_config('$n', $2, false);/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^']+)\b$n\s+IS NOT NULL/$1current_setting('$n') != ''/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^']+)\b$n\s+IS NULL/$1current_setting('$n') = ''/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^']+)\b$n\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^\.']+)\b$self->{global_variables}{$n}{name}\s+IS NOT NULL/$1current_setting('$n') != ''/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^\.']+)\b$self->{global_variables}{$n}{name}\s+IS NULL/$1current_setting('$n') = ''/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{code} =~ s/([^\.']+)\b$self->{global_variables}{$n}{name}\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; # Replace global variable in DECLARE section too $i = 0; while ($fct_detail{declare} =~ s/([^']+)\b$n\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; $i = 0; while ($fct_detail{declare} =~ s/([^\.']+)\b$self->{global_variables}{$n}{name}\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; } # Replace call to raise exception foreach my $e (keys %{$self->{custom_exception}}) { $fct_detail{code} =~ s/\bRAISE\s+$e\b/RAISE EXCEPTION '$e' USING ERRCODE = '$self->{custom_exception}{$e}'/igs; $fct_detail{code} =~ s/(\s+(?:WHEN|OR)\s+)$e\s+/$1SQLSTATE '$self->{custom_exception}{$e}' /igs; } # Remove %ROWTYPE from return type $fct_detail{func_ret_type} =~ s/\%ROWTYPE//igs; if ($self->{json_test} && !$meta && ($self->{type} eq uc($fct_detail{type}) || $self->{type} eq 'PACKAGE')) { my $dirprefix = ''; $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); my $tfh = $self->append_export_file($dirprefix . "$self->{type}.json", 1); flock($tfh, 2) || die "FATAL: can't lock file ${dirprefix}$self->{type}.json\n"; $tfh->print($self->{json_config}); $self->close_export_file($tfh, 1); } return %fct_detail; } sub _list_all_functions { my $self = shift; my $oraver = ''; # OWNER|OBJECT_NAME|PROCEDURE_NAME|OBJECT_TYPE my $sql = qq{ SELECT p.owner,p.object_name,p.procedure_name,o.object_type FROM $self->{prefix}_PROCEDURES p JOIN $self->{prefix}_OBJECTS o ON p.owner = o.owner AND p.object_name = o.object_name WHERE o.object_type IN ('PROCEDURE','PACKAGE','FUNCTION') AND o.TEMPORARY='N' AND o.GENERATED='N' AND o.SECONDARY='N' }; if ($self->{db_version} =~ /Release 8/) { $sql = qq{ SELECT p.owner,p.object_name,p.procedure_name,o.object_type FROM $self->{prefix}_PROCEDURES p, $self->{prefix}_OBJECTS o WHERE o.object_type IN ('PROCEDURE','PACKAGE','FUNCTION') AND p.owner = o.owner AND p.object_name = o.object_name AND o.TEMPORARY='N' AND o.GENERATED='N' AND o.SECONDARY='N' }; } if (!$self->{export_invalid}) { $sql .= " AND o.STATUS = 'VALID'"; } elsif ($self->{export_invalid} == 2) { $sql .= " AND o.STATUS <> 'VALID'"; } if ($self->{schema}) { $sql .= " AND p.OWNER='$self->{schema}'"; } else { $sql .= " AND p.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } my @infos = (); my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { next if (($row[3] eq 'PACKAGE') && !$row[2]); if ( $row[2] ) { # package_name.fct_name push(@infos, lc("$row[1].$row[2]")); } elsif ( $self->{export_schema} ) { # package_name.fct_name push(@infos, lc("$row[0].$row[1]")); } else { # owner.fct_name push(@infos, lc($row[1])); } } $sth->finish(); return @infos; } sub _sql_type { my ($self, $type, $len, $precision, $scale, $default, $no_blob_to_oid) = @_; my $data_type = ''; chomp($type); # Simplify timestamp type if (uc($type) eq 'TIMESTAMP') { $len = '' if ($len >= 6); $type = "timestamp($len)" if ($len); } # Replace SYS_REFCURSOR as Pg REFCURSOR $type =~ s/SYS_REFCURSOR/refcursor/i; # Interval precision for year/month/day is not supported by PostgreSQL if ($type =~ /INTERVAL/) { $type =~ s/(INTERVAL\s+YEAR)\s*\(\d+\)/$1/; $type =~ s/(INTERVAL\s+YEAR\s+TO\s+MONTH)\s*\(\d+\)/$1/; $type =~ s/(INTERVAL\s+DAY)\s*\(\d+\)/$1/; # maximum precision allowed for seconds is 6 if ($type =~ /INTERVAL\s+DAY\s+TO\s+SECOND\s*\((\d+)\)/) { if ($1 > 6) { $type =~ s/(INTERVAL\s+DAY\s+TO\s+SECOND)\s*\(\d+\)/$1(6)/; } } } # Overide the length if ( ($type eq 'NUMBER') && $precision ) { $len = $precision; return $self->{data_type}{'NUMBER(*)'} if ($scale eq '0' && exists $self->{data_type}{'NUMBER(*)'}); return $self->{data_type}{"NUMBER(*,$scale)"} if (exists $self->{data_type}{"NUMBER(*,$scale)"}); } elsif ( ($type eq 'NUMBER') && ($len == 38) ) { if ($scale eq '0' && $precision eq '') { # Allow custom type rewrite for NUMBER(*,0) return $self->{data_type}{'NUMBER(*,0)'} if (exists $self->{data_type}{'NUMBER(*,0)'}); } $precision = $len; } elsif ( $type =~ /CHAR/ && $len && exists $self->{data_type}{"$type($len)"}) { return $self->{data_type}{"$type($len)"}; } elsif ( $type =~ /RAW/ ) { $self->{use_uuid} = 1 if (($len && exists $self->{data_type}{"$type($len)"}) || ($default =~ /(SYS_GUID|$self->{uuid_function})/i)); return $self->{data_type}{"$type($len)"} if ($len && exists $self->{data_type}{"$type($len)"}); return 'uuid' if ($default =~ /(SYS_GUID|$self->{uuid_function})/i); } elsif ($type =~ /BLOB/ && $self->{blob_to_lo} && !$no_blob_to_oid) { # we want to convert BLOB into large object return 'oid'; } # Special case of * precision if ($precision eq '*') { if ($len ne '*') { $precision = $len; } else { $precision = 38; } } # handing timestamp with local timezone my $captured_type = ''; if($type =~ /TIMESTAMP\s*\((\d+)\) WITH LOCAL TIME ZONE/i){ $type =~ s/TIMESTAMP\s*\((\d+)\) WITH LOCAL TIME ZONE/TIMESTAMP WITH LOCAL TIME ZONE/ig; $captured_type = $1; my $res_type = $self->{data_type}{$type}; my @res_ = split(/\s+/,$res_type); $res_[0] .= "($captured_type)"; my $r_type = join(' ',@res_); $type = uc($r_type); } if (exists $self->{data_type}{$type}) { $len *= 2 if ($len > 0 && $self->{double_max_varchar} && $type =~ /VARCHAR/); if ($len) { if ( ($type eq "CHAR") || ($type eq "NCHAR") || ($type =~ /VARCHAR/) ) { # Type CHAR have default length set to 1 # Type VARCHAR(2) must have a specified length $len = 1 if (!$len && (($type eq "CHAR") || ($type eq "NCHAR")) ); return "$self->{data_type}{$type}($len)"; } elsif ($type eq "NUMBER") { # This is an integer if (!$scale) { if ($precision) { if (exists $self->{data_type}{"$type($precision)"}) { return $self->{data_type}{"$type($precision)"}; } if ($self->{pg_integer_type}) { if ($precision < 5) { return 'smallint'; } elsif ($precision <= 9) { return 'integer'; # The speediest in PG } elsif ($precision <= 19) { return 'bigint'; } else { return "numeric($precision)"; } } return "numeric($precision)"; } elsif ($self->{pg_integer_type}) { # For number without precision default is to use bigint # but mark the column for review (#) if it needs to be # translated into numeric instead my $need_review = ''; $need_review = '#' if ($self->{type} eq 'SHOW_COLUMN'); return $self->{default_numeric} || 'bigint' . $need_review; } } else { if (exists $self->{data_type}{"$type($precision,$scale)"}) { return $self->{data_type}{"$type($precision,$scale)"}; } if ($self->{pg_numeric_type}) { if ($precision eq '') { return "decimal(38, $scale)"; } if ($precision >= $scale) { if ($precision <= 6) { if ($self->{pg_supports_negative_scale}) { return "decimal($precision,$scale)"; } else { return 'real'; } } elsif ($precision <= 15) { return 'double precision'; } } } $precision = 38 if ($precision eq ''); if ($scale > $precision) { return "numeric"; } return "decimal($precision,$scale)"; } } return "$self->{data_type}{$type}"; } else { if (($type eq 'NUMBER') && $self->{pg_integer_type}) { # For number without precision default is to use bigint # but mark the column for review (#) if it needs to be # translated into numeric instead my $need_review = ''; $need_review = '#' if ($self->{type} eq 'SHOW_COLUMN'); return $self->{default_numeric} . $need_review; } else { return $self->{data_type}{$type}; } } } return $type; } sub _get_job { my($self) = @_; # Jobs appears in version 10 only return if ($self->{db_version} =~ /Release [8|9]/); # Retrieve all database job from user_jobs table my $str = "SELECT JOB,WHAT,INTERVAL,SCHEMA_USER FROM $self->{prefix}_JOBS"; if (!$self->{schema}) { $str .= " WHERE SCHEMA_USER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE SCHEMA_USER = '$self->{schema}'"; } $str .= $self->limit_to_objects('JOB', 'JOB'); #$str .= " ORDER BY JOB"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[3].$row->[0]"; } $data{$row->[0]}{what} = $row->[1]; $data{$row->[0]}{interval} = $row->[2]; } # Retrieve all database jobs from view [ALL|DBA]_SCHEDULER_JOBS $str = "SELECT job_name AS JOB, job_action AS WHAT, repeat_interval AS INTERVAL, owner AS SCHEMA_USER"; $str .= " FROM $self->{prefix}_SCHEDULER_JOBS"; $str .= " WHERE repeat_interval IS NOT NULL"; $str .= " AND client_id IS NULL"; if (!$self->{schema}) { $str .= " AND owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND owner = '$self->{schema}'"; } $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[3].$row->[0]"; } $data{$row->[0]}{what} = $row->[1]; $data{$row->[0]}{interval} = $row->[2]; } return %data; } sub _get_dblink { my($self) = @_; # Retrieve all database link from dba_db_links table my $str = "SELECT OWNER,DB_LINK,USERNAME,HOST FROM $self->{prefix}_DB_LINKS"; if (!$self->{schema}) { $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE OWNER = '$self->{schema}'"; } $str .= $self->limit_to_objects('DBLINK', 'DB_LINK'); #$str .= " ORDER BY DB_LINK"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } $data{$row->[1]}{owner} = $row->[0]; $data{$row->[1]}{user} = $row->[2]; $data{$row->[1]}{username} = $self->{pg_user} || $row->[2]; $data{$row->[1]}{host} = $row->[3]; } return %data; } =head2 _get_partitions This function implements an Oracle-native partitions information. Return two hash ref with partition details and partition default. =cut sub _get_partitions { my ($self) = @_; my $highvalue = 'A.HIGH_VALUE'; if ($self->{db_version} =~ /Release 8/) { $highvalue = "'' AS HIGH_VALUE"; } my $ref_constraint = "'' AS REF_PTN_CONSTRAINT_NAME"; if ($self->{db_version} =~ /Release (2|3|19)/) { $ref_constraint = 'B.REF_PTN_CONSTRAINT_NAME'; } my $condition = ''; if ($self->{schema}) { $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; } else { $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Retrieve all partitions. my $str = qq{ SELECT A.TABLE_NAME, A.PARTITION_POSITION, A.PARTITION_NAME, $highvalue, A.TABLESPACE_NAME, B.PARTITIONING_TYPE, C.NAME, C.COLUMN_NAME, C.COLUMN_POSITION, A.TABLE_OWNER, $ref_constraint FROM $self->{prefix}_TAB_PARTITIONS A, $self->{prefix}_PART_TABLES B, $self->{prefix}_PART_KEY_COLUMNS C WHERE a.table_name = b.table_name AND (b.partitioning_type = 'RANGE' OR b.partitioning_type = 'LIST' OR b.partitioning_type = 'HASH' OR b.partitioning_type = 'REFERENCE') AND a.table_name = c.name $condition }; if ($self->{db_version} !~ /Release 8/) { $str .= $self->exclude_mviews('A.TABLE_OWNER, A.TABLE_NAME'); } $str .= $self->limit_to_objects('TABLE|PARTITION', 'A.TABLE_NAME|A.PARTITION_NAME'); if ($self->{prefix} ne 'USER') { if ($self->{schema}) { $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } else { $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } } $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_POSITION,C.COLUMN_POSITION\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); my %default = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[9].$row->[0]"; } if ( ($row->[3] eq 'DEFAULT')) { $default{$row->[0]}{name} = $row->[2]; $default{$row->[0]}{tablespace} = $row->[4]; next; } $parts{$row->[0]}{$row->[1]}{name} = $row->[2]; my %refinfo = (); if ($row->[10]) { %refinfo = _get_ref_key_info($self, $row->[10], $row->[9]); $refinfo{refconstraint} = $row->[10]; if ($self->{partition_by_reference} eq 'duplicate') { $row->[5] = $refinfo{refparttype}; } elsif ($self->{partition_by_reference} =~ /^\d+$/) { $row->[5] = 'HASH'; } } push(@{$parts{$row->[0]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[7], 'colpos' => $row->[8], 'tablespace' => $row->[4], 'owner' => $row->[9], %refinfo }); } $sth->finish; return \%parts, \%default; } =head2 _get_subpartitions This function implements a MySQL subpartitions information. Return two hash ref with partition details and partition default. =cut sub _get_subpartitions { my($self) = @_; my $highvalue = 'A.HIGH_VALUE'; if ($self->{db_version} =~ /Release [89]/) { $highvalue = "'' AS HIGH_VALUE"; } my $condition = ''; if ($self->{schema}) { $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; } else { $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Retrieve all partitions. my $str = qq{ SELECT A.TABLE_NAME, A.SUBPARTITION_POSITION, A.SUBPARTITION_NAME, $highvalue, A.TABLESPACE_NAME, B.SUBPARTITIONING_TYPE, C.NAME, C.COLUMN_NAME, C.COLUMN_POSITION, A.TABLE_OWNER, A.PARTITION_NAME FROM $self->{prefix}_tab_subpartitions A, $self->{prefix}_part_tables B, $self->{prefix}_subpart_key_columns C WHERE a.table_name = b.table_name AND (b.subpartitioning_type = 'RANGE' OR b.subpartitioning_type = 'LIST' OR b.subpartitioning_type = 'HASH' OR b.subpartitioning_type = 'REFERENCE') AND a.table_name = c.name $condition }; $str .= $self->limit_to_objects('TABLE|PARTITION', 'A.TABLE_NAME|A.SUBPARTITION_NAME'); if ($self->{prefix} ne 'USER') { if ($self->{schema}) { $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } else { $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } } if ($self->{db_version} !~ /Release 8/) { $str .= $self->exclude_mviews('A.TABLE_OWNER, A.TABLE_NAME'); } $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION,C.COLUMN_POSITION\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %subparts = (); my %default = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[9].$row->[0]"; } if ( ($row->[3] eq 'MAXVALUE') || ($row->[3] eq 'DEFAULT')) { $default{$row->[0]}{$row->[10]}{name} = $row->[2]; $default{$row->[0]}{$row->[10]}{tablespace} = $row->[4]; next; } $subparts{$row->[0]}{$row->[10]}{$row->[1]}{name} = $row->[2]; push(@{$subparts{$row->[0]}{$row->[10]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[7], 'colpos' => $row->[8], 'tablespace' => $row->[4], 'owner' => $row->[9]}); } $sth->finish; return \%subparts, \%default; } =head2 _get_partitions_list This function implements a MySQL-native partitions information. Return a hash of the partition table_name => type =cut sub _get_partitions_list { my ($self) = @_; my $highvalue = 'A.HIGH_VALUE'; if ($self->{db_version} =~ /Release [89]/) { $highvalue = "'' AS HIGH_VALUE"; } my $condition = ''; if ($self->{schema}) { $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; } else { $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Retrieve all partitions. my $str = qq{ SELECT A.TABLE_NAME, A.PARTITION_POSITION, A.PARTITION_NAME, $highvalue, A.TABLESPACE_NAME, B.PARTITIONING_TYPE, A.TABLE_OWNER FROM $self->{prefix}_TAB_PARTITIONS A, $self->{prefix}_PART_TABLES B WHERE A.TABLE_NAME = B.TABLE_NAME $condition }; if ($self->{db_version} !~ /Release 8/) { $str .= $self->exclude_mviews('A.TABLE_OWNER, A.TABLE_NAME'); } $str .= $self->limit_to_objects('TABLE|PARTITION','A.TABLE_NAME|A.PARTITION_NAME'); if ($self->{prefix} ne 'USER') { if ($self->{schema}) { $str .= "\tAND A.TABLE_OWNER ='$self->{schema}'\n"; } else { $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')\n"; } } #$str .= "ORDER BY A.TABLE_NAME\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { $parts{$row->[5]}++; } $sth->finish; return %parts; } =head2 _get_partitioned_table Return a hash of the partitioned table with the number of partition =cut sub _get_partitioned_table { my($self, %subpart) = @_; my $highvalue = 'A.HIGH_VALUE'; if ($self->{db_version} =~ /Release [89]/) { $highvalue = "'' AS HIGH_VALUE"; } my $ref_constraint = "'' AS REF_PTN_CONSTRAINT_NAME"; if ($self->{db_version} =~ /Release (2|3|19)/) { $ref_constraint = 'B.REF_PTN_CONSTRAINT_NAME'; } my $condition = ''; if ($self->{schema}) { $condition .= "AND B.OWNER='$self->{schema}' "; } else { $condition .= " AND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Retrieve all partitions. my $str = "SELECT B.TABLE_NAME, B.PARTITIONING_TYPE, B.OWNER, B.PARTITION_COUNT, B.SUBPARTITIONING_TYPE"; if ($self->{type} !~ /SHOW|TEST/) { $str .= ", C.COLUMN_NAME, C.COLUMN_POSITION, $ref_constraint"; $str .= " FROM $self->{prefix}_PART_TABLES B, $self->{prefix}_PART_KEY_COLUMNS C"; $str .= " WHERE B.TABLE_NAME = C.NAME AND (B.PARTITIONING_TYPE = 'RANGE' OR B.PARTITIONING_TYPE = 'LIST' OR B.PARTITIONING_TYPE = 'HASH' OR B.PARTITIONING_TYPE = 'REFERENCE')"; } else { $str .= " FROM $self->{prefix}_PART_TABLES B WHERE (B.PARTITIONING_TYPE = 'RANGE' OR B.PARTITIONING_TYPE = 'LIST' OR B.PARTITIONING_TYPE = 'HASH' OR B.PARTITIONING_TYPE = 'REFERENCE') AND B.SUBPARTITIONING_TYPE <> 'SYSTEM' "; } $str .= $self->limit_to_objects('TABLE','B.TABLE_NAME'); if ($self->{prefix} ne 'USER') { if ($self->{type} !~ /SHOW|TEST/) { if ($self->{schema}) { $str .= "\tAND B.OWNER ='$self->{schema}' AND C.OWNER=B.OWNER\n"; } else { $str .= "\tAND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=C.OWNER\n"; } } else { if ($self->{schema}) { $str .= "\tAND B.OWNER ='$self->{schema}'\n"; } else { $str .= "\tAND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')\n"; } } } if ($self->{db_version} !~ /Release 8/) { $str .= $self->exclude_mviews('B.OWNER, B.TABLE_NAME'); } if ($self->{type} !~ /SHOW|TEST/) { $str .= "ORDER BY B.OWNER,B.TABLE_NAME,C.COLUMN_POSITION\n"; } else { $str .= "ORDER BY B.OWNER,B.TABLE_NAME\n"; } my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[2].$row->[0]"; } # when this is not a composite partition the count is defined # when this is not the default number of subpartition $parts{"\L$row->[0]\E"}{count} = 0; $parts{"\L$row->[0]\E"}{composite} = 0; if (exists $subpart{"\L$row->[0]\E"}) { $parts{"\L$row->[0]\E"}{composite} = 1; foreach my $k (keys %{$subpart{"\L$row->[0]\E"}}) { $parts{"\L$row->[0]\E"}{count} += $subpart{"\L$row->[0]\E"}{$k}{count}; } $parts{"\L$row->[0]\E"}{count} = $row->[3] if (!$parts{"\L$row->[0]\E"}{count}); } else { $parts{"\L$row->[0]\E"}{count} = $row->[3]; } $parts{"\L$row->[0]\E"}{type} = $row->[1]; if ($row->[7]) { $parts{"\L$row->[0]\E"}{refconstraint} = $row->[7]; my %refinfo = _get_ref_key_info($self, $row->[7], $row->[2]); $parts{"\L$row->[0]\E"}{reftable} = $refinfo{reftable}; $parts{"\L$row->[0]\E"}{refrtable} = $refinfo{refrtable}; $parts{"\L$row->[0]\E"}{refcolumn} = $refinfo{refcolumn}; $parts{"\L$row->[0]\E"}{reftype} = $refinfo{reftype}; $parts{"\L$row->[0]\E"}{reflength} = $refinfo{reflength}; $parts{"\L$row->[0]\E"}{refprecision} = $refinfo{refprecision}; $parts{"\L$row->[0]\E"}{refscale} = $refinfo{refscale}; $parts{"\L$row->[0]\E"}{refcharlengh} = $refinfo{refcharlengh}; $parts{"\L$row->[0]\E"}{refparttype} = $refinfo{refparttype}; if ($self->{partition_by_reference} eq 'duplicate') { $parts{"\L$row->[0]\E"}{type} = $refinfo{refparttype}; } elsif ($self->{partition_by_reference} =~ /^\d+$/) { $parts{"\L$row->[0]\E"}{type} = 'HASH'; } } if ($self->{type} !~ /SHOW|TEST/) { push(@{ $parts{"\L$row->[0]\E"}{columns} }, $row->[5]); } } $sth->finish; return %parts; } =head2 _get_objects This function retrieves all object the Oracle information =cut sub _get_objects { my $self = shift; my $temporary = "TEMPORARY='N'"; if ($self->{export_gtt} or $self->{type} =~ /^(COPY|INSERT)$/) { $temporary = "(TEMPORARY='N' OR OBJECT_TYPE='TABLE')"; } my $oraver = ''; # OWNER|OBJECT_NAME|SUBOBJECT_NAME|OBJECT_ID|DATA_OBJECT_ID|OBJECT_TYPE|CREATED|LAST_DDL_TIME|TIMESTAMP|STATUS|TEMPORARY|GENERATED|SECONDARY my $sql = "SELECT OBJECT_NAME,OBJECT_TYPE,STATUS FROM $self->{prefix}_OBJECTS WHERE $temporary AND GENERATED='N' AND SECONDARY='N' AND OBJECT_TYPE <> 'SYNONYM' AND OBJECT_NAME NOT LIKE 'BIN\$%'"; if ($self->{schema}) { $sql .= " AND OWNER='$self->{schema}'"; } else { $sql .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } my %infos = (); my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; my %count = (); while ( my @row = $sth->fetchrow()) { my $valid = ($row[2] eq 'VALID') ? 0 : 1; push(@{$infos{$row[1]}}, { ( name => $row[0], invalid => $valid ) }); $count{$row[1]}{$valid}++; } $sth->finish(); if ($self->{debug}) { foreach my $k (sort keys %count) { print STDERR "\tFound $count{$k}{0} valid and ", ($count{$k}{1}||0), " invalid object $k\n"; } } return %infos; } sub _get_privilege { my($self) = @_; my %privs = (); my %roles = (); # Retrieve all privilege per table defined in this database my $str = "SELECT b.GRANTEE,b.OWNER,b.TABLE_NAME,b.PRIVILEGE,a.OBJECT_TYPE,b.GRANTABLE FROM DBA_TAB_PRIVS b, DBA_OBJECTS a"; if ($self->{schema}) { $str .= " WHERE b.GRANTOR = '$self->{schema}'"; } else { $str .= " WHERE b.GRANTOR NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $str .= " AND b.TABLE_NAME=a.OBJECT_NAME AND a.OWNER=b.GRANTOR"; if ($self->{grant_object} && $self->{grant_object} ne 'USER') { $str .= " AND a.OBJECT_TYPE = '\U$self->{grant_object}\E'"; } else { $str .= " AND a.OBJECT_TYPE <> 'TYPE'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'b.GRANTEE|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME'); if (!$self->{export_invalid}) { $str .= " AND a.STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND a.STATUS <> 'VALID'"; } #$str .= " ORDER BY b.TABLE_NAME, b.GRANTEE"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { next if ($row->[0] eq 'PUBLIC'); if (!$self->{schema} && $self->{export_schema}) { $row->[2] = "$row->[1].$row->[2]"; } $privs{$row->[2]}{type} = $row->[4]; $privs{$row->[2]}{owner} = $row->[1] if (!$privs{$row->[2]}{owner}); if ($row->[5] eq 'YES') { $privs{$row->[2]}{grantable} = $row->[5]; } push(@{$privs{$row->[2]}{privilege}{$row->[0]}}, $row->[3]); push(@{$roles{owner}}, $row->[1]) if (!grep(/^$row->[1]$/, @{$roles{owner}})); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); # Retrieve all privilege per column table defined in this database $str = "SELECT b.GRANTEE,b.OWNER,b.TABLE_NAME,b.PRIVILEGE,b.COLUMN_NAME FROM DBA_COL_PRIVS b, DBA_OBJECTS a"; if ($self->{schema}) { $str .= " WHERE b.GRANTOR = '$self->{schema}'"; } else { $str .= " WHERE b.GRANTOR NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } if (!$self->{export_invalid}) { $str .= " AND a.STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND a.STATUS <> 'VALID'"; } $str .= " AND b.TABLE_NAME=a.OBJECT_NAME AND a.OWNER=b.GRANTOR AND a.OBJECT_TYPE <> 'TYPE'"; if ($self->{grant_object} && $self->{grant_object} ne 'USER') { $str .= " AND a.OBJECT_TYPE = '\U$self->{grant_object}\E'"; } else { $str .= " AND a.OBJECT_TYPE <> 'TYPE'"; } $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'b.GRANTEE|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME'); $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[2] = "$row->[1].$row->[2]"; } $privs{$row->[2]}{owner} = $row->[1] if (!$privs{$row->[2]}{owner}); push(@{$privs{$row->[2]}{column}{$row->[4]}{$row->[0]}}, $row->[3]); push(@{$roles{owner}}, $row->[1]) if (!grep(/^$row->[1]$/, @{$roles{owner}})); push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); } $sth->finish(); # Search if users have admin rights my @done = (); foreach my $r (@{$roles{owner}}, @{$roles{grantee}}) { next if (grep(/^$r$/, @done)); push(@done, $r); # Get all system priviledge given to a role $str = "SELECT PRIVILEGE,ADMIN_OPTION FROM DBA_SYS_PRIVS WHERE GRANTEE = '$r'"; $str .= " " . $self->limit_to_objects('GRANT', 'GRANTEE'); #$str .= " ORDER BY PRIVILEGE"; $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { push(@{$roles{admin}{$r}{privilege}}, $row->[0]); push(@{$roles{admin}{$r}{admin_option}}, $row->[1]); } $sth->finish(); } # Now try to find if it's a user or a role foreach my $u (@done) { $str = "SELECT GRANTED_ROLE FROM DBA_ROLE_PRIVS WHERE GRANTEE = '$u'"; $str .= " " . $self->limit_to_objects('GRANT', 'GRANTEE'); #$str .= " ORDER BY GRANTED_ROLE"; $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { push(@{$roles{role}{$u}}, $row->[0]); } $str = "SELECT USERNAME FROM DBA_USERS WHERE USERNAME = '$u'"; $str .= " " . $self->limit_to_objects('GRANT', 'USERNAME'); #$str .= " ORDER BY USERNAME"; $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $roles{type}{$u} = 'USER'; } next if $roles{type}{$u}; $str = "SELECT ROLE,PASSWORD_REQUIRED FROM DBA_ROLES WHERE ROLE='$u'"; $str .= " " . $self->limit_to_objects('GRANT', 'ROLE'); #$str .= " ORDER BY ROLE"; $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { $roles{type}{$u} = 'ROLE'; $roles{password_required}{$u} = $row->[1]; } $sth->finish(); } return (\%privs, \%roles); } =head2 _get_database_size This function retrieves the size of the MySQL database in MB =cut sub _get_database_size { my $self = shift; my $mb_size = ''; my $sql = "SELECT sum(bytes)/1024/1024 FROM USER_SEGMENTS"; if (!$self->{user_grants}) { $sql = "SELECT sum(bytes)/1024/1024 FROM DBA_SEGMENTS"; if ($self->{schema}) { $sql .= " WHERE OWNER='$self->{schema}' "; } else { $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } } my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute or return undef; while ( my @row = $sth->fetchrow()) { $mb_size = sprintf("%.2f MB", $row[0]); last; } $sth->finish(); return $mb_size; } =head2 _get_largest_tables This function retrieves the list of largest table of the Oracle database in MB =cut sub _get_largest_tables { my $self = shift; my %table_size = (); my $prefix = 'USER'; my $owner_segment = ''; $owner_segment = " AND A.OWNER='$self->{schema}'"; if (!$self->{user_grants}) { $prefix = 'DBA'; $owner_segment = ' AND S.OWNER=A.OWNER'; } my $sql = "SELECT * FROM ( SELECT S.SEGMENT_NAME, ROUND(S.BYTES/1024/1024) SIZE_MB FROM ${prefix}_SEGMENTS S JOIN $self->{prefix}_TABLES A ON (S.SEGMENT_NAME=A.TABLE_NAME$owner_segment) WHERE S.SEGMENT_TYPE LIKE 'TABLE%' AND A.SECONDARY = 'N'"; if ($self->{db_version} =~ /Release 8/) { $sql = "SELECT * FROM ( SELECT A.SEGMENT_NAME, ROUND(A.BYTES/1024/1024) SIZE_MB FROM ${prefix}_SEGMENTS A WHERE A.SEGMENT_TYPE LIKE 'TABLE%'"; } if ($self->{db_version} !~ /Release 8/ || !$self->{user_grants}) { if ($self->{schema}) { $sql .= " AND A.OWNER='$self->{schema}'"; } else { $sql .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } } if ($self->{db_version} =~ /Release 8/) { $sql .= $self->limit_to_objects('TABLE', 'A.SEGMENT_NAME'); } else { $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); } if ($self->{db_version} =~ /Release 8/) { $sql .= " ORDER BY A.BYTES DESC, A.SEGMENT_NAME ASC) WHERE ROWNUM <= $self->{top_max}"; } else { $sql .= " ORDER BY S.BYTES DESC, S.SEGMENT_NAME ASC) WHERE ROWNUM <= $self->{top_max}"; } my $sth = $self->{dbh}->prepare( $sql ) or return undef; $sth->execute(@{$self->{query_bind_params}}) or return undef; while ( my @row = $sth->fetchrow()) { $table_size{$row[0]} = $row[1]; } $sth->finish(); return %table_size; } sub _get_audit_queries { my($self) = @_; my @users = (); push(@users, split(/[,;\s]/, uc($self->{audit_user}))); # Retrieve all object with tablespaces. my $str = "SELECT SQL_TEXT FROM DBA_AUDIT_TRAIL WHERE ACTION_NAME IN ('INSERT','UPDATE','DELETE','SELECT')"; if (($#users >= 0) && !grep(/^ALL$/, @users)) { $str .= " AND USERNAME IN ('" . join("','", @users) . "')"; } my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tmp_queries = (); while (my $row = $sth->fetch) { $self->_remove_comments(\$row->[0], 1); $self->{comment_values} = (); $row->[0] =~ s/\%ORA2PG_COMMENT\d+\%//gs; $row->[0] = $self->normalize_query($row->[0]); $tmp_queries{$row->[0]}++; } $sth->finish; my %queries = (); my $i = 1; foreach my $q (keys %tmp_queries) { $queries{$i} = $q; $i++; } return %queries; } sub _get_synonyms { my ($self) = shift; # Retrieve all synonym my $str = "SELECT OWNER,SYNONYM_NAME,TABLE_OWNER,TABLE_NAME,DB_LINK FROM $self->{prefix}_SYNONYMS"; if ($self->{schema}) { $str .= " WHERE owner='$self->{schema}' AND table_owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } else { $str .= " WHERE owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND table_owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } $str .= $self->limit_to_objects('SYNONYM','SYNONYM_NAME'); #$str .= " ORDER BY SYNONYM_NAME\n"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %synonyms = (); while (my $row = $sth->fetch) { next if ($row->[1] =~ /^\//); # Some not fully deleted synonym start with a slash if (!$self->{schema} && $self->{export_schema}) { $row->[1] = $row->[0] . '.' . $row->[1]; } $synonyms{$row->[1]}{owner} = $row->[0]; $synonyms{$row->[1]}{table_owner} = $row->[2]; $synonyms{$row->[1]}{table_name} = $row->[3]; $synonyms{$row->[1]}{dblink} = $row->[4]; } $sth->finish; return %synonyms; } sub _get_tablespaces { my ($self) = shift; # Retrieve all object with tablespaces. my $str = qq{ SELECT a.SEGMENT_NAME,a.TABLESPACE_NAME,a.SEGMENT_TYPE,c.FILE_NAME, a.OWNER, a.PARTITION_NAME, b.SUBOBJECT_NAME FROM DBA_SEGMENTS a, $self->{prefix}_OBJECTS b, DBA_DATA_FILES c WHERE a.SEGMENT_TYPE IN ('INDEX', 'TABLE', 'TABLE PARTITION') AND a.SEGMENT_NAME = b.OBJECT_NAME AND a.SEGMENT_TYPE = b.OBJECT_TYPE AND a.OWNER = b.OWNER AND a.TABLESPACE_NAME = c.TABLESPACE_NAME }; if ($self->{schema}) { $str .= " AND a.OWNER='$self->{schema}'"; } else { $str .= " AND a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $str .= $self->limit_to_objects('TABLESPACE|TABLE', 'a.TABLESPACE_NAME|a.SEGMENT_NAME'); $str .= " ORDER BY a.SEGMENT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tbs = (); my @done = (); while (my $row = $sth->fetch) { my $table = $row->[0]; if ($self->{export_schema} && !$self->{schema}) { $row->[0] = "$row->[4].$row->[0]"; } next if (grep(/^$row->[0]$/, @done)); push(@done, $row->[0]); push(@{$tbs{$row->[2]}{$row->[1]}{$row->[3]}}, $row->[0]); # With partitioned table, add tablespace info for table partition if (exists $self->{partitions}{$table}) { foreach my $pos (sort {$self->{partitions}{$table}{$a} <=> $self->{partitions}{$table}{$b}} keys %{$self->{partitions}{$table}}) { my $part_name = $self->{partitions}{$table}{$pos}{name}; my $tbpart_name = $part_name; $tbpart_name = $table . '_part' . $pos if ($self->{rename_partition}); next if ($self->{allow_partition} && !grep($_ =~ /^$tbpart_name$/i, @{$self->{allow_partition}})); my $tbspace = $self->{partitions}{$table}{$pos}{info}[0]->{tablespace}; push(@{$tbs{$row->[2]}{$tbspace}{$row->[3]}}, $tbpart_name); if (exists $self->{subpartitions}{$table}{$part_name}) { foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) { my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); my $sub_tb_name = $subpart; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any $sub_tb_name = $table . '_part' . $pos . '_subpart' . $p if ($self->{rename_partition}); if (exists $self->{tables}{$table}{field_name} && $#{$self->{tables}{$table}{field_name}} < 0) { $self->logit("Table $table has no column defined, skipping...\n", 1); next; } my $tbspace = $self->{subpartitions}{$table}{$part_name}{$p}{info}[0]->{tablespace}; push(@{$tbs{$row->[2]}{$tbspace}{$row->[3]}}, $sub_tb_name); } # process default subpartition table if (exists $self->{subpartitions_default}{$table}{$part_name}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}{name}$/i, @{$self->{allow_partition}})) { my $sub_tb_name = $self->{subpartitions_default}{$table}{$part_name}{name}; $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any next if ($self->{allow_partition} && !grep($_ =~ /^$sub_tb_name$/i, @{$self->{allow_partition}})); $sub_tb_name = $table . '_part' . $pos . '_subpart_default' if ($self->{rename_partition}); my $tbspace = $self->{subpartitions_default}{$table}{$part_name}{tablespace}; push(@{$tbs{$row->[2]}{$tbspace}{$row->[3]}}, $sub_tb_name); } } } } # Add the default partition table if (exists $self->{partitions_default}{$table}) { if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}{name}$/i, @{$self->{allow_partition}})) { my $tbpart_name = $table . '_' . $self->{partitions_default}{$table}{name}; $tbpart_name = $table . '_part_default' if ($self->{rename_partition}); my $tbspace = $self->{partitions_default}{$table}{tablespace}; push(@{$tbs{$row->[2]}{$tbspace}{$row->[3]}}, $tbpart_name); } } } } $sth->finish; return \%tbs; } sub _list_tablespaces { my ($self) = shift; # list tablespaces. my $str = qq{ SELECT c.FILE_NAME, c.TABLESPACE_NAME, a.OWNER, ROUND(c.BYTES/1024000) MB FROM DBA_DATA_FILES c, DBA_SEGMENTS a WHERE a.TABLESPACE_NAME = c.TABLESPACE_NAME }; if ($self->{schema}) { $str .= " AND a.OWNER='$self->{schema}'"; } else { $str .= " AND a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $str .= $self->limit_to_objects('TABLESPACE', 'c.TABLESPACE_NAME'); #$str .= " ORDER BY c.TABLESPACE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tbs = (); while (my $row = $sth->fetch) { $tbs{$row->[1]}{path} = $row->[0]; $tbs{$row->[1]}{owner} = $row->[2]; } $sth->finish; return \%tbs; } sub _get_sequences { my ($self) = shift; # Retrieve all indexes my $str = "SELECT DISTINCT SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, LAST_NUMBER, CACHE_SIZE, CYCLE_FLAG, SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; if (!$self->{schema}) { $str .= " WHERE SEQUENCE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE SEQUENCE_OWNER = '$self->{schema}'"; } # Exclude sequence used for IDENTITY columns $str .= " AND SEQUENCE_NAME NOT LIKE 'ISEQ\$\$_%'"; $str .= $self->limit_to_objects($self->{type}, 'SEQUENCE_NAME'); #$str .= " ORDER BY SEQUENCE_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %seqs = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = $row->[7] . '.' . $row->[0]; } push(@{$seqs{$row->[0]}}, @$row); } return \%seqs; } sub _extract_sequence_info { my ($self) = shift; my $sql = "SELECT DISTINCT SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, CYCLE_FLAG, ORDER_FLAG, CACHE_SIZE, LAST_NUMBER,SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; if ($self->{schema}) { $sql .= " WHERE SEQUENCE_OWNER='$self->{schema}'"; } else { $sql .= " WHERE SEQUENCE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } $sql .= $self->limit_to_objects('SEQUENCE','SEQUENCE_NAME'); my @script = (); my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr ."\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $seq_info = $sth->fetchrow_hashref) { my $seqname = $seq_info->{SEQUENCE_NAME}; if (!$self->{schema} && $self->{export_schema}) { $seqname = $seq_info->{SEQUENCE_OWNER} . '.' . $seq_info->{SEQUENCE_NAME}; } my $nextvalue = $seq_info->{LAST_NUMBER} + $seq_info->{INCREMENT_BY}; my $alter = "ALTER SEQUENCE $self->{pg_supports_ifexists} " . $self->quote_object_name($seqname) . " RESTART WITH $nextvalue;"; push(@script, $alter); $self->logit("Extracted sequence information for sequence \"$seqname\"\n", 1); } $sth->finish(); return @script; } sub _column_attributes { my ($self, $table, $owner, $objtype) = @_; $objtype ||= 'TABLE'; my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND A.OWNER='$owner' "; } else { $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } my $sql = qq{SELECT A.COLUMN_NAME, A.NULLABLE, A.DATA_DEFAULT, A.TABLE_NAME, A.OWNER, A.COLUMN_ID, A.DATA_TYPE FROM $self->{prefix}_TAB_COLUMNS A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' and A.TABLE_NAME NOT LIKE 'BIN\$%' $condition ORDER BY A.COLUMN_ID}; if ($self->{db_version} =~ /Release 8/) { # an 8i database. $sql = qq{SELECT A.COLUMN_NAME, A.NULLABLE, A.DATA_DEFAULT, A.TABLE_NAME, A.OWNER, A.COLUMN_ID, A.DATA_TYPE FROM $self->{prefix}_TAB_COLUMNS A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' and A.TABLE_NAME NOT LIKE 'BIN\$%' $condition ORDER BY A.COLUMN_ID}; } my $sth = $self->{dbh}->prepare($sql); if (!$sth) { $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { my $spatial_srid = 0; if ($self->{export_schema} && !$self->{schema}) { $data{"$row->[4].$row->[3]"}{"$row->[0]"}{nullable} = $row->[1]; $data{"$row->[4].$row->[3]"}{"$row->[0]"}{default} = $row->[2]; # Store the data type of the column following its position $data{"$row->[4].$row->[3]"}{data_type}{$row->[5]} = $row->[6]; } else { $data{$row->[3]}{"$row->[0]"}{nullable} = $row->[1]; $data{$row->[3]}{"$row->[0]"}{default} = $row->[2]; # Store the data type of the column following its position $data{$row->[3]}{data_type}{$row->[5]} = $row->[6]; } my $f = $self->{tables}{"$table"}{column_info}{"$row->[0]"}; if ( ($f->[1] =~ /SDO_GEOMETRY/i) && ($self->{convert_srid} <= 1) ) { $spatial_srid = "SELECT COALESCE(SRID, $self->{default_srid}) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME='\U$table\E' AND COLUMN_NAME='$row->[0]' AND OWNER='\U$self->{tables}{$table}{table_info}{owner}\E'"; if ($self->{convert_srid} == 1) { $spatial_srid = "SELECT COALESCE(sdo_cs.map_oracle_srid_to_epsg(SRID), $self->{default_srid}) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME='\U$table\E' AND COLUMN_NAME='$row->[0]' AND OWNER='\U$self->{tables}{$table}{table_info}{owner}\E'"; } my $sth2 = $self->{dbh}->prepare($spatial_srid); if (!$sth2) { if ($self->{dbh}->errstr !~ /ORA-01741/) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } else { # No SRID defined, use default one $spatial_srid = $self->{default_srid} || '0'; $self->logit("WARNING: Error retreiving SRID, no matter default SRID will be used: $spatial_srid\n", 0); } } else { $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @result = (); while (my $r = $sth2->fetch) { push(@result, $r->[0]) if ($r->[0] =~ /\d+/); } $sth2->finish(); if ($self->{export_schema} && !$self->{schema}) { $data{"$row->[4].$row->[3]"}{"$row->[0]"}{spatial_srid} = $result[0] || $self->{default_srid} || '0'; } else { $data{$row->[3]}{"$row->[0]"}{spatial_srid} = $result[0] || $self->{default_srid} || '0'; } } } } return %data; } sub _list_triggers { my($self) = @_; # Retrieve all indexes my $str = "SELECT T.TRIGGER_NAME, T.TABLE_NAME, T.OWNER FROM $self->{prefix}_TRIGGERS T JOIN $self->{prefix}_OBJECTS O ON (T.TRIGGER_NAME = O.OBJECT_NAME AND T.OWNER = O.OWNER) WHERE O.OBJECT_TYPE = 'TRIGGER'"; if (!$self->{export_invalid}) { $str .= " AND O.STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND O.STATUS <> 'VALID'"; } if (!$self->{schema}) { $str .= " AND T.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND T.OWNER = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','T.TABLE_NAME|T.TABLE_NAME|T.TRIGGER_NAME'); #$str .= " ORDER BY TABLE_NAME, TRIGGER_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %triggers = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { push(@{$triggers{"$row->[2].$row->[1]"}}, $row->[0]); } else { push(@{$triggers{$row->[1]}}, $row->[0]); } } return %triggers; } sub _global_temp_table_info { my($self) = @_; my $owner = ''; if ($self->{schema}) { $owner .= "AND A.OWNER='$self->{schema}' "; } else { $owner .= "AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Get comment on global temporary table my %comments = (); if ($self->{type} eq 'TABLE') { my $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM $self->{prefix}_TAB_COMMENTS A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='TABLE' $owner"; if ($self->{db_version} !~ /Release 8/) { $sql .= $self->exclude_mviews('A.OWNER, A.TABLE_NAME'); } $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[3].$row->[0]"; } $comments{$row->[0]}{comment} = $row->[1]; $comments{$row->[0]}{table_type} = $row->[2]; } $sth->finish(); } my $sql = "SELECT A.OWNER,A.TABLE_NAME,NVL(num_rows,1) NUMBER_ROWS,A.TABLESPACE_NAME,A.NESTED,A.LOGGING,A.DURATION FROM $self->{prefix}_TABLES A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER AND A.TABLE_NAME=O.OBJECT_NAME AND O.OBJECT_TYPE='TABLE' $owner"; $sql .= " AND A.TEMPORARY='Y'"; if ($self->{db_version} !~ /Release [89]/) { $sql .= " AND (A.DROPPED IS NULL OR A.DROPPED = 'NO')"; } $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); $sql .= " AND (A.IOT_TYPE IS NULL OR A.IOT_TYPE = 'IOT')"; #$sql .= " ORDER BY A.OWNER, A.TABLE_NAME"; my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %tables_infos = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } $tables_infos{$row->[1]}{owner} = $row->[0] || ''; $tables_infos{$row->[1]}{num_rows} = $row->[2] || 0; $tables_infos{$row->[1]}{tablespace} = $row->[3] || 0; $tables_infos{$row->[1]}{comment} = $comments{$row->[1]}{comment} || ''; $tables_infos{$row->[1]}{type} = $comments{$row->[1]}{table_type} || ''; $tables_infos{$row->[1]}{nested} = $row->[4] || ''; if ($row->[5] eq 'NO') { $tables_infos{$row->[1]}{nologging} = 1; } else { $tables_infos{$row->[1]}{nologging} = 0; } $tables_infos{$row->[1]}{num_rows} = 0; $tables_infos{$row->[1]}{temporary} = 'Y'; $tables_infos{$row->[1]}{duration} = $row->[6]; } $sth->finish(); return %tables_infos; } sub _encrypted_columns { my ($self, $table, $owner) = @_; # Encryption appears in version 10 only return if ($self->{db_version} =~ /Release [8|9]/); my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND A.OWNER='$owner' "; } else { $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } $condition =~ s/^\s*AND /WHERE /s; my $sth = $self->{dbh}->prepare(<{prefix}_ENCRYPTED_COLUMNS A $condition END if (!$sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $data{"$row->[2].$row->[1].$row->[0]"} = $row->[3]; } else { $data{"$row->[1].$row->[0]"} = $row->[3]; } } return %data; } sub _get_subpartitioned_table { my($self) = @_; my $highvalue = 'A.HIGH_VALUE'; if ($self->{db_version} =~ /Release [89]/) { $highvalue = "'' AS HIGH_VALUE"; } my $condition = ''; if ($self->{schema}) { $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; } else { $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } # Retrieve all partitions. my $str = "SELECT A.TABLE_NAME, A.PARTITION_NAME, A.SUBPARTITION_NAME, A.SUBPARTITION_POSITION, B.SUBPARTITIONING_TYPE, A.TABLE_OWNER, B.PARTITION_COUNT"; if ($self->{type} !~ /SHOW|TEST/) { $str .= ", C.COLUMN_NAME, C.COLUMN_POSITION"; $str .= " FROM $self->{prefix}_TAB_SUBPARTITIONS A, $self->{prefix}_PART_TABLES B, $self->{prefix}_SUBPART_KEY_COLUMNS C"; } else { $str .= " FROM $self->{prefix}_TAB_SUBPARTITIONS A, $self->{prefix}_PART_TABLES B"; } $str .= " WHERE A.TABLE_NAME = B.TABLE_NAME AND (B.SUBPARTITIONING_TYPE = 'RANGE' OR B.SUBPARTITIONING_TYPE = 'LIST' OR B.SUBPARTITIONING_TYPE = 'HASH' OR B.SUBPARTITIONING_TYPE = 'REFERENCE')"; $str .= " AND A.TABLE_NAME = C.NAME" if ($self->{type} !~ /SHOW|TEST/); $str .= $self->limit_to_objects('TABLE|PARTITION','A.TABLE_NAME|A.PARTITION_NAME'); if ($self->{prefix} ne 'USER') { if ($self->{type} !~ /SHOW|TEST/) { if ($self->{schema}) { $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } else { $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; } } else { if ($self->{schema}) { $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER\n"; } else { $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER\n"; } } } if ($self->{db_version} !~ /Release 8/) { $str .= $self->exclude_mviews('A.TABLE_OWNER, A.TABLE_NAME'); } if ($self->{type} !~ /SHOW|TEST/) { $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION,C.COLUMN_POSITION\n"; } else { $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION\n"; } my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %parts = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[5].$row->[0]"; } $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{type} = $row->[4]; $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{count}++; push(@{ $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{columns} }, $row->[7]) if (!grep(/^$row->[7]$/, @{ $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{columns} })); } $sth->finish; return %parts; } sub _get_plsql_metadata { my $self = shift; my $owner = shift; # Retrieve all functions my $str = "SELECT DISTINCT OBJECT_NAME,OWNER,OBJECT_TYPE FROM $self->{prefix}_OBJECTS WHERE (OBJECT_TYPE = 'FUNCTION' OR OBJECT_TYPE = 'PROCEDURE' OR OBJECT_TYPE = 'PACKAGE BODY')"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if ($owner) { $str .= " AND OWNER = '$owner'"; $self->logit("Looking forward functions declaration in schema $owner.\n", 1) if (!$self->{quiet}); } elsif (!$self->{schema}) { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; $self->logit("Looking forward functions declaration in all schema.\n", 1) if (!$self->{quiet}); } else { $str .= " AND OWNER = '$self->{schema}'"; $self->logit("Looking forward functions declaration in schema $self->{schema}.\n", 1) if (!$self->{quiet}); } #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %functions = (); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); while (my $row = $sth->fetch) { next if (grep(/^$row->[1].$row->[0]$/i, @fct_done)); push(@fct_done, "$row->[1].$row->[0]"); $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}{type} = $row->[2]; } $sth->finish(); # Get content of package body my $sql = "SELECT NAME, OWNER, TYPE, TEXT FROM $self->{prefix}_SOURCE"; if ($owner) { $sql .= " WHERE OWNER = '$owner'"; } elsif (!$self->{schema}) { $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " WHERE OWNER = '$self->{schema}'"; } $sql .= " AND TYPE <> 'PACKAGE'"; $sql .= " ORDER BY OWNER, NAME, LINE"; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { next if (!exists $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}); $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}{text} .= $row->[3]; } $sth->finish(); # For each schema in the Oracle instance foreach my $sch (sort keys %{ $self->{function_metadata} }) { next if ( ($owner && ($sch ne $owner)) || (!$owner && $self->{schema} && ($sch ne $self->{schema})) ); # Look for functions/procedures foreach my $name (sort keys %{$self->{function_metadata}{$sch}{'none'}}) { if ($self->{function_metadata}{$sch}{'none'}{$name}{type} ne 'PACKAGE BODY') { # Retrieve metadata for this function after removing comments $self->_remove_comments(\$self->{function_metadata}{$sch}{'none'}{$name}{text}, 1); $self->{comment_values} = (); $self->{function_metadata}{$sch}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; my %fct_detail = $self->_lookup_function($self->{function_metadata}{$sch}{'none'}{$name}{text}, undef, 1); if (!exists $fct_detail{name}) { delete $self->{function_metadata}{$sch}{'none'}{$name}; next; } delete $fct_detail{code}; delete $fct_detail{before}; %{$self->{function_metadata}{$sch}{'none'}{$name}{metadata}} = %fct_detail; delete $self->{function_metadata}{$sch}{'none'}{$name}{text}; } else { $self->_remove_comments(\$self->{function_metadata}{$sch}{'none'}{$name}{text}, 1); $self->{comment_values} = (); $self->{function_metadata}{$sch}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; my %infos = $self->_lookup_package($self->{function_metadata}{$sch}{'none'}{$name}{text}); delete $self->{function_metadata}{$sch}{'none'}{$name}; $name =~ s/"//g; foreach my $f (sort keys %infos) { next if (!$f); my $fn = lc($f); delete $infos{$f}{code}; delete $infos{$f}{before}; %{$self->{function_metadata}{$sch}{$name}{$fn}{metadata}} = %{$infos{$f}}; my $res_name = $f; $res_name =~ s/^([^\.]+)\.//; $f =~ s/^([^\.]+)\.//; if ($self->{package_as_schema}) { $res_name = $name . '.' . $res_name; } else { $res_name = $name . '_' . $res_name; } $res_name =~ s/"_"/_/g; $f =~ s/"//g; $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{name} = $self->quote_object_name($res_name); $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{package} = $name; } } } } } sub _get_security_definer { my ($self, $type) = @_; my %security = (); # This table does not exists before 10g return if ($self->{db_version} =~ /Release [89]/); # Retrieve security privilege per function defined in this database # Version of Oracle 10 does not have the OBJECT_TYPE column. my $str = "SELECT AUTHID,OBJECT_TYPE,OBJECT_NAME,OWNER FROM $self->{prefix}_PROCEDURES"; if ($self->{db_version} =~ /Release 10/) { $str = "SELECT AUTHID,'ALL' AS OBJECT_TYPE,OBJECT_NAME,OWNER FROM $self->{prefix}_PROCEDURES"; } if ($self->{schema}) { $str .= " WHERE OWNER = '$self->{schema}'"; } else { $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } if ( $type && ($self->{db_version} !~ /Release 10/) ) { $str .= " AND OBJECT_TYPE='$type'"; } $str .= " " . $self->limit_to_objects('FUNCTION|PROCEDURE|PACKAGE|TRIGGER', 'OBJECT_NAME|OBJECT_NAME|OBJECT_NAME|OBJECT_NAME'); #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { next if (!$row->[0]); if (!$self->{schema} && $self->{export_schema}) { $row->[2] = "$row->[3].$row->[2]"; } $security{$row->[2]}{security} = $row->[0]; $security{$row->[2]}{owner} = $row->[3]; } $sth->finish(); return (\%security); } =head2 _get_identities This function retrieve information about IDENTITY columns that must be exported as PostgreSQL serial. =cut sub _get_identities { my ($self) = @_; # Identity column appears in version 12 only return if ($self->{db_version} =~ /Release (8|9|10|11)/); # Retrieve all indexes my $str = "SELECT OWNER, TABLE_NAME, COLUMN_NAME, GENERATION_TYPE, IDENTITY_OPTIONS FROM $self->{prefix}_TAB_IDENTITY_COLS"; if (!$self->{schema}) { $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE OWNER = '$self->{schema}'"; } $str .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); #$str .= " ORDER BY OWNER, TABLE_NAME, COLUMN_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %seqs = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[1] = "$row->[0].$row->[1]"; } # GENERATION_TYPE can be ALWAYS, BY DEFAULT and BY DEFAULT ON NULL $seqs{$row->[1]}{$row->[2]}{generation} = $row->[3]; # SEQUENCE options $seqs{$row->[1]}{$row->[2]}{options} = $row->[4]; $seqs{$row->[1]}{$row->[2]}{options} =~ s/(SCALE|EXTEND|SESSION)_FLAG: .//ig; $seqs{$row->[1]}{$row->[2]}{options} =~ s/KEEP_VALUE: .//is; $seqs{$row->[1]}{$row->[2]}{options} =~ s/(START WITH):/$1/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/(INCREMENT BY):/$1/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/MAX_VALUE:/MAXVALUE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/MIN_VALUE:/MINVALUE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/CYCLE_FLAG: N/NO CYCLE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/CYCLE_FLAG: Y/CYCLE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE_SIZE:/CACHE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE_SIZE:/CACHE/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/ORDER_FLAG: .//; $seqs{$row->[1]}{$row->[2]}{options} =~ s/,//g; $seqs{$row->[1]}{$row->[2]}{options} =~ s/\s$//; $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE\s+0/CACHE 1/; # For default values don't use option at all if ( $seqs{$row->[1]}{$row->[2]}{options} eq 'START WITH 1 INCREMENT BY 1 MAXVALUE 9999999999999999999999999999 MINVALUE 1 NO CYCLE CACHE 20') { delete $seqs{$row->[1]}{$row->[2]}{options}; } # Limit the sequence value to bigint max $seqs{$row->[1]}{$row->[2]}{options} =~ s/MAXVALUE 9999999999999999999999999999/MAXVALUE 9223372036854775807/; $seqs{$row->[1]}{$row->[2]}{options} =~ s/\s+/ /g; } return %seqs; } =head2 _get_materialized_views This function implements a mysql-native materialized views information. Returns a hash of view names with the SQL queries they are based on. =cut sub _get_materialized_views { my($self) = @_; # Retrieve all views my $str = "SELECT MVIEW_NAME,QUERY,UPDATABLE,REFRESH_MODE,REFRESH_METHOD,USE_NO_INDEX,REWRITE_ENABLED,BUILD_MODE,OWNER FROM $self->{prefix}_MVIEWS"; if ($self->{db_version} =~ /Release 8/) { $str = "SELECT MVIEW_NAME,QUERY,UPDATABLE,REFRESH_MODE,REFRESH_METHOD,'',REWRITE_ENABLED,BUILD_MODE,OWNER FROM $self->{prefix}_MVIEWS"; } if (!$self->{schema}) { $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE OWNER = '$self->{schema}'"; } $str .= $self->limit_to_objects('MVIEW', 'MVIEW_NAME'); #$str .= " ORDER BY MVIEW_NAME"; my $sth = $self->{dbh}->prepare($str); if (not defined $sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } if (not $sth->execute(@{$self->{query_bind_params}})) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); return (); } my %data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[8].$row->[0]"; } $data{$row->[0]}{text} = $row->[1]; $data{$row->[0]}{updatable} = ($row->[2] eq 'Y') ? 1 : 0; $data{$row->[0]}{refresh_mode} = $row->[3]; $data{$row->[0]}{refresh_method} = $row->[4]; $data{$row->[0]}{no_index} = ($row->[5] eq 'Y') ? 1 : 0; $data{$row->[0]}{rewritable} = ($row->[6] eq 'Y') ? 1 : 0; $data{$row->[0]}{build_mode} = $row->[7]; $data{$row->[0]}{owner} = $row->[8]; } return %data; } sub _get_materialized_view_names { my($self) = @_; # Retrieve all views my $str = "SELECT MVIEW_NAME,OWNER FROM $self->{prefix}_MVIEWS"; if (!$self->{schema}) { $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " WHERE OWNER = '$self->{schema}'"; } $str .= $self->limit_to_objects('MVIEW', 'MVIEW_NAME'); #$str .= " ORDER BY MVIEW_NAME"; my $sth = $self->{dbh}->prepare($str); if (not defined $sth) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } if (not $sth->execute(@{$self->{query_bind_params}})) { $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); } my @data = (); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } push(@data, uc($row->[0])); } return @data; } sub _get_package_function_list { my ($self, $owner) = @_; # Retrieve all package information my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE = 'PACKAGE BODY'"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if ($owner) { $str .= " AND OWNER = '$owner'"; $self->logit("Looking forward functions declaration in schema $owner.\n", 1) if (!$self->{quiet}); } elsif (!$self->{schema}) { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; $self->logit("Looking forward functions declaration in all schema.\n", 1) if (!$self->{quiet}); } else { $str .= " AND OWNER = '$self->{schema}'"; $self->logit("Looking forward functions declaration in schema $self->{schema}.\n", 1) if (!$self->{quiet}); } #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my @packages = (); while (my $row = $sth->fetch) { next if (grep(/^$row->[0]$/i, @packages)); push(@packages, $row->[0]); } $sth->finish(); # Get content of all packages definition my $sql = "SELECT NAME, OWNER, TYPE, TEXT FROM $self->{prefix}_SOURCE"; if ($owner) { $sql .= " WHERE OWNER = '$owner'"; } elsif (!$self->{schema}) { $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " WHERE OWNER = '$self->{schema}'"; } $sql .= " AND TYPE <> 'PACKAGE'"; $sql .= " ORDER BY OWNER, NAME, LINE"; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); my %function_metadata = (); my $has_pkg = $#packages; while (my $row = $sth->fetch) { next if ($has_pkg >= 0 && !grep(/^$row->[0]$/, @packages)); $function_metadata{$row->[1]}{$row->[0]}{text} .= $row->[3]; } $sth->finish(); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); foreach my $sch (sort keys %function_metadata) { next if ( ($owner && ($sch ne $owner)) || (!$owner && $self->{schema} && ($sch ne $self->{schema})) ); foreach my $name (sort keys %{$function_metadata{$sch}}) { $self->_remove_comments(\$function_metadata{$sch}{$name}{text}, 1); $self->{comment_values} = (); $function_metadata{$sch}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; my %infos = $self->_lookup_package($function_metadata{$sch}{$name}{text}); delete $function_metadata{$sch}{$name}; foreach my $f (sort keys %infos) { next if (!$f); my $fn = lc($f); my $res_name = $f; if ($res_name =~ s/^([^\.]+)\.//) { next if (lc($1) ne lc($name)); } if ($self->{package_as_schema}) { $res_name = $name . '.' . $res_name; } else { $res_name = $name . '_' . $res_name; } $res_name =~ s/"_"/_/g; $f =~ s/"//gs; if ($res_name) { $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{name} = $self->quote_object_name($res_name); $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{package} = $name; } } } } } sub _get_procedures { my ($self) = @_; # Retrieve all functions my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='PROCEDURE'"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if (!$self->{schema}) { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND OWNER = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('PROCEDURE','OBJECT_NAME'); #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %procedures = (); my @fct_done = (); push(@fct_done, @EXCLUDED_FUNCTION); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } next if (grep(/^$row->[0]$/i, @fct_done)); push(@fct_done, $row->[0]); $procedures{"$row->[0]"}{owner} = $row->[1]; } $sth->finish(); my $sql = "SELECT NAME,OWNER,TEXT FROM $self->{prefix}_SOURCE"; if (!$self->{schema}) { $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $sql .= " WHERE OWNER = '$self->{schema}'"; } $sql .= " " . $self->limit_to_objects('PROCEDURE','NAME'); $sql .= " ORDER BY OWNER,NAME,LINE"; $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); while (my $row = $sth->fetch) { if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } # Fix possible Malformed UTF-8 character $row->[2] = encode('UTF-8', $row->[2]) if ($self->{force_plsql_encoding}); # Remove some bargage when migrating from 8i $row->[2] =~ s/\bAUTHID\s+[^\s]+\s+//is; if (exists $procedures{"$row->[0]"}) { $procedures{"$row->[0]"}{text} .= $row->[2]; } } return \%procedures; } sub _get_packages { my ($self) = @_; # Retrieve the list of packages my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE = 'PACKAGE'"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if (!$self->{schema}) { $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; } else { $str .= " AND OWNER = '$self->{schema}'"; } $str .= " " . $self->limit_to_objects('PACKAGE','OBJECT_NAME'); #$str .= " ORDER BY OBJECT_NAME"; my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); my %packages = (); my @fct_done = (); while (my $row = $sth->fetch) { $self->logit("\tFound Package: $row->[0]\n", 1); next if (grep(/^$row->[0]$/, @fct_done)); push(@fct_done, $row->[0]); # Get package definition first my $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND TYPE='PACKAGE' ORDER BY LINE"; my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { $packages{$row->[0]}{desc} .= 'CREATE OR REPLACE ' if ($r->[0] =~ /^PACKAGE\s+/is); $packages{$row->[0]}{desc} .= $r->[0]; } $sth2->finish(); $packages{$row->[0]}{desc} .= "\n" if (exists $packages{$row->[0]}); # Then package body code $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND TYPE='PACKAGE BODY' ORDER BY LINE"; $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { $packages{$row->[0]}{text} .= 'CREATE OR REPLACE ' if ($r->[0] =~ /^PACKAGE\s+/is); $packages{$row->[0]}{text} .= $r->[0]; } $packages{$row->[0]}{owner} = $row->[1]; } return \%packages; } sub _get_types { my ($self, $name) = @_; # Retrieve all user defined types my $str = "SELECT DISTINCT OBJECT_NAME,OWNER,OBJECT_ID FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='TYPE'"; if (!$self->{export_invalid}) { $str .= " AND STATUS='VALID'"; } elsif ($self->{export_invalid} == 2) { $str .= " AND STATUS <> 'VALID'"; } if ($name) { $str .= " AND OBJECT_NAME='$name'"; } else { $str .= " AND OBJECT_NAME NOT LIKE 'SYS_PLSQL_%'"; # found in export from 9i } $str .= " AND GENERATED='N'"; if ($self->{schema}) { $str .= "AND OWNER='$self->{schema}' "; } else { $str .= "AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$name) { $str .= $self->limit_to_objects('TYPE', 'OBJECT_NAME'); } else { @{$self->{query_bind_params}} = (); } #$str .= " ORDER BY OBJECT_NAME"; # use a separeate connection my $local_dbh = _db_connection($self); my $sth = $local_dbh->prepare($str) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); my @types = (); my @fct_done = (); while (my $row = $sth->fetch) { next if ($row->[0] =~ /^(SDO_GEOMETRY|ST_|STGEOM_)/); #my $sql = "SELECT DBMS_METADATA.GET_DDL('TYPE','$row->[0]','$row->[1]') FROM DUAL"; my $sql = "SELECT TEXT,LINE FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND (TYPE='TYPE' OR TYPE='TYPE BODY') ORDER BY TYPE, LINE"; if (!$self->{schema} && $self->{export_schema}) { $row->[0] = "$row->[1].$row->[0]"; } $self->logit("\tFound Type: $row->[0]\n", 1); next if (grep(/^$row->[0]$/, @fct_done)); push(@fct_done, $row->[0]); my %tmp = (); my $sth2 = $local_dbh->prepare($sql) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); while (my $r = $sth2->fetch) { $tmp{code} .= $r->[0]; } $sth2->finish(); $tmp{name} = $row->[0]; $tmp{owner} = $row->[1]; $tmp{pos} = $row->[2]; if (!$self->{preserve_case}) { $tmp{code} =~ s/(TYPE\s+)"[^"]+"\."[^"]+"/$1\L$row->[0]\E/igs; $tmp{code} =~ s/(TYPE\s+)"[^"]+"/$1\L$row->[0]\E/igs; } else { $tmp{code} =~ s/((?:CREATE|REPLACE|ALTER)\s+TYPE\s+)([^"\s]+)\s/$1"$2" /igs; } $tmp{code} =~ s/\s+ALTER/;\nALTER/igs; push(@types, \%tmp); } $sth->finish(); $local_dbh->disconnect() if ($local_dbh); return \@types; } sub _col_count { my ($self, $table, $owner) = @_; my $condition = ''; $condition .= "AND A.TABLE_NAME='$table' " if ($table); if ($owner) { $condition .= "AND A.OWNER='$owner' "; } else { $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; } if (!$table) { $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); } else { @{$self->{query_bind_params}} = (); } my $sth = ''; if ($self->{db_version} !~ /Release 8/) { $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='TABLE' and A.TABLE_NAME NOT LIKE 'BIN\$%' $condition GROUP BY A.OWNER, A.TABLE_NAME END if (!$sth) { $self->logit("FATAL: _col_count() " . $self->{dbh}->errstr . "\n", 0, 1); } } else { # an 8i database. $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, $self->{prefix}_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='TABLE' and A.TABLE_NAME NOT LIKE 'BIN\$%' $condition GROUP BY A.OWNER, A.TABLE_NAME END if (!$sth) { $self->logit("FATAL: _col_count() " . $self->{dbh}->errstr . "\n", 0, 1); } } $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); my %data = (); while (my $row = $sth->fetch) { if ($self->{export_schema} && !$self->{schema}) { $data{"$row->[0].$row->[1]"} = $row->[2]; } else { $data{$row->[1]} = $row->[2]; } } return %data; } =head2 auto_set_encoding This function is used to find the PostgreSQL charset corresponding to the Oracle NLS_LANG value =cut sub auto_set_encoding { my $oracle_charset = shift; my %ENCODING = ( "AL32UTF8" => "UTF8", "JA16EUC" => "EUC_JP", "JA16SJIS" => "EUC_JIS_2004", "ZHT32EUC" => "EUC_TW", "CL8ISO8859P5" => "ISO_8859_5", "AR8ISO8859P6" => "ISO_8859_6", "EL8ISO8859P7" => "ISO_8859_7", "IW8ISO8859P8" => "ISO_8859_8", "CL8KOI8R" => "KOI8R", "CL8KOI8U" => "KOI8U", "WE8ISO8859P1" => "LATIN1", "EE8ISO8859P2" => "LATIN2", "SE8ISO8859P3" => "LATIN3", "NEE8ISO8859P4"=> "LATIN4", "WE8ISO8859P9" => "LATIN5", "NE8ISO8859P10"=> "LATIN6", "BLT8ISO8859P13"=> "LATIN7", "CEL8ISO8859P14"=> "LATIN8", "WE8ISO8859P15" => "LATIN9", "RU8PC866" => "WIN866", "EE8MSWIN1250" => "WIN1250", "CL8MSWIN1251" => "WIN1251", "WE8MSWIN1252" => "WIN1252", "EL8MSWIN1253" => "WIN1253", "TR8MSWIN1254" => "WIN1254", "IW8MSWIN1255" => "WIN1255", "AR8MSWIN1256" => "WIN1256", "BLT8MSWIN1257"=> "WIN1257" ); foreach my $k (keys %ENCODING) { return $ENCODING{$k} if (uc($oracle_charset) eq $k); } return ''; } sub _has_dbms_log_execute_privilege { my $self = shift; my $has_dbms_log_execute_privilege = 0; my $sql = "SELECT CASE WHEN COUNT(*) > 0 THEN 1 ELSE 0 END AS has_execute_privilege FROM USER_TAB_PRIVS WHERE TABLE_NAME = 'DBMS_LOB' AND PRIVILEGE = 'EXECUTE'"; my $sth = $self->{dbh}->prepare( $sql ) or return 0; $sth->execute or return 0; while ( my @row = $sth->fetchrow()) { $has_dbms_log_execute_privilege = $row[0]; } $sth->finish(); $self->logit("DEBUG: Source database user " . (!$has_dbms_log_execute_privilege ? "doesn't ": "") . "have 'EXECUTE' privilege on 'DBMS_LOG'\n", 1); return $has_dbms_log_execute_privilege; } 1; ora2pg-25.0/lib/Ora2Pg/PLSQL.pm000066400000000000000000004617671500113072400157410ustar00rootroot00000000000000package Ora2Pg::PLSQL; #------------------------------------------------------------------------------ # Project : Oracle to PostgreSQL database schema converter # Name : Ora2Pg/PLSQL.pm # Language : Perl # Authors : Gilles Darold, gilles _AT_ darold _DOT_ net # Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved - # Function : Perl module used to convert Oracle PLSQL code into PL/PGSQL # Usage : See documentation #------------------------------------------------------------------------------ # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see < http://www.gnu.org/licenses/ >. # #------------------------------------------------------------------------------ use vars qw($VERSION %OBJECT_SCORE $SIZE_SCORE $FCT_TEST_SCORE $QUERY_TEST_SCORE %UNCOVERED_SCORE %UNCOVERED_MYSQL_SCORE @ORA_FUNCTIONS @MYSQL_SPATIAL_FCT @MYSQL_FUNCTIONS %EXCEPTION_MAP %MAX_SCORE %MSSQL_STYLE %UNCOVERED_MSSQL_SCORE); use POSIX qw(locale_h); #set locale to LC_NUMERIC C setlocale(LC_NUMERIC,"C"); $VERSION = '25.0'; #---------------------------------------------------- # Cost scores used when converting PLSQL to PLPGSQL #---------------------------------------------------- # Scores associated to each database objects: %OBJECT_SCORE = ( 'CLUSTER' => 0, # Not supported and no equivalent 'FUNCTION' => 1, # read/adapt the header 'INDEX' => 0.1, # Read/adapt - use varcharops like operator ? 'FUNCTION-BASED-INDEX' => 0.2, # Check code of function call 'REV-INDEX' => 1, # Check/rewrite the index to use trigram 'CHECK' => 0.1, # Check/adapt the check constraint 'MATERIALIZED VIEW' => 3, # Read/adapt, will just concern automatic snapshot export 'PACKAGE BODY' => 3, # Look at globals variables and global 'PROCEDURE' => 1, # read/adapt the header 'SEQUENCE' => 0.1, # read/adapt to convert name.nextval() into nextval('name') 'TABLE' => 0.1, # read/adapt the column type/name 'TABLE PARTITION' => 0.1, # Read/check that table partitionning is ok 'TABLE SUBPARTITION' => 0.2, # Read/check that table sub partitionning is ok 'TRIGGER' => 1, # read/adapt the header 'TYPE' => 1, # read 'TYPE BODY' => 10, # Not directly supported need adaptation 'VIEW' => 1, # read/adapt 'DATABASE LINK' => 3, # Supported as FDW using oracle_fdw 'GLOBAL TEMPORARY TABLE' => 10, # supported, but not permanent in PostgreSQL 'DIMENSION' => 0, # Not supported and no equivalent 'JOB' => 2, # read/adapt 'SYNONYM' => 0.1, # read/adapt 'QUERY' => 0.2, # read/adapt 'ENCRYPTED COLUMN' => 20, ## adapt using pg_crypto ); # Max score to applicate per type of object %MAX_SCORE = ( 'INDEX' => 288, # 3 man days 'SEQUENCE' => 288, # 3 man days 'TABLE' => 672, # 7 man days 'TABLE PARTITION' => 480, # 5 man days 'TABLE SUBPARTITION' => 480, # 5 man days 'GLOBAL TEMPORARY TABLE' => 288, # 3 man days 'SYNONYM' => 192, # 2 man days ); # Scores following the number of characters: 1000 chars for one unit. # Note: his correspond to the global read time not to the difficulty. $SIZE_SCORE = 1000; # Cost to apply on each function or query for testing $FCT_TEST_SCORE = 2; $QUERY_TEST_SCORE = 0.1; # Scores associated to each code difficulties. %UNCOVERED_SCORE = ( 'TRUNC' => 0.1, 'IS TABLE OF' => 4, 'OUTER JOIN' => 2, 'CONNECT BY' => 3, 'BULK COLLECT' => 5, 'GOTO' => 2, 'FORALL' => 1, 'ROWNUM' => 1, 'NOTFOUND' => 0, 'ISOPEN' => 1, 'ROWCOUNT' => 1, 'ROWID' => 2, 'UROWID' => 2, 'IS RECORD' => 1, 'SQLCODE' => 1, 'TABLE' => 2, 'DBMS_' => 3, 'DBMS_OUTPUT.put' => 1, 'UTL_' => 3, 'CTX_' => 3, 'EXTRACT' => 0.1, 'EXCEPTION' => 2, 'TO_NUMBER' => 0.1, 'REGEXP_LIKE' => 0.1, 'REGEXP_COUNT' => 0.2, 'REGEXP_INSTR' => 1, 'REGEXP_SUBSTR' => 1, 'TG_OP' => 0, 'CURSOR' => 1, 'PIPE ROW' => 1, 'ORA_ROWSCN' => 3, 'SAVEPOINT' => 1, 'DBLINK' => 1, 'PLVDATE' => 2, 'PLVSTR' => 2, 'PLVCHR' => 2, 'PLVSUBST' => 2, 'PLVLEX' => 2, 'PLUNIT' => 2, 'ADD_MONTHS' => 0.1, 'LAST_DAY' => 1, 'NEXT_DAY' => 1, 'MONTHS_BETWEEN' => 1, 'SDO_' => 3, 'PRAGMA' => 3, 'MDSYS' => 1, 'MERGE INTO' => 3, 'COMMIT' => 1, 'CONTAINS' => 1, 'SCORE' => 1, 'FUZZY' => 1, 'NEAR' => 1, 'TO_CHAR' => 0.1, 'TO_NCHAR' => 0.1, 'ANYDATA' => 2, 'CONCAT' => 0.1, 'TIMEZONE' => 1, 'JSON' => 3, 'TO_CLOB' => 0.1, 'XMLTYPE' => 3, 'CREATENONSCHEMABASEDXML' => 3, 'CREATESCHEMABASEDXML' => 3, 'CREATEXML' => 3, 'EXISTSNODE' => 3, 'EXTRACT' => 3, 'GETNAMESPACE' => 3, 'GETROOTELEMENT' => 3, 'GETSCHEMAURL' => 3, 'ISFRAGMENT' => 3, 'ISSCHEMABASED' => 3, 'ISSCHEMAVALID' => 3, 'ISSCHEMAVALIDATED' => 3, 'SCHEMAVALIDATE' => 3, 'SETSCHEMAVALIDATED' => 3, 'TOOBJECT' => 3, 'TRANSFORM' => 3, 'FND_CONC_GLOBAL' => 3, 'FND_CONCURRENT' => 3, 'FND_FILE' => 1, 'FND_PROGRAM' => 3, 'FND_SET' => 3, 'FND_REQUEST' => 3, 'FND_REQUEST_INFO' => 3, 'FND_SUBMIT' => 3, 'FND_GLOBAL' => 1, 'FND_PROFILE' => 1, 'FND_CURRENCY' => 3, 'FND_ORG' => 3, 'FND_STANDARD' => 3, 'FND_UTILITIES' => 3, 'ADD CONSTRAINT' => 3, ## need stability in constraint name 'HTP' => 0.2, "'SSSSS'" => 2, # adapt with extract(epoch from $1::time) if not a timestamp "'J'" => 2, # adapt with orafce to_date 'WHEN OTHER' => 10, # This make lot of bug undetectable ); @ORA_FUNCTIONS = qw( AsciiStr Compose Decompose Dump VSize Bin_To_Num CharToRowid HexToRaw NumToDSInterval NumToYMInterval RawToHex To_Clob To_DSInterval To_Lob To_Multi_Byte To_NClob To_Single_Byte To_YMInterval BFilename Cardinality Group_ID LNNVL NANVL Sys_Context Uid UserEnv Bin_To_Num BitAnd Median Remainder DbTimeZone New_Time SessionTimeZone Tz_Offset Get_Env From_Tz ); @MYSQL_SPATIAL_FCT = ( 'AsBinary', 'AsText', 'Buffer', 'Centroid', 'Contains', 'Crosses', 'Dimension', 'Disjoint', 'EndPoint', 'Envelope', 'Equals', 'ExteriorRing', 'GeomCollFromText', 'GeomCollFromWKB', 'GeometryN', 'GeometryType', 'GeomFromText', 'GeomFromWKB', 'GLength', 'InteriorRingN', 'Intersects', 'IsClosed', 'IsSimple', 'LineFromText', 'LineFromWKB', 'MLineFromText', 'MPointFromText', 'MPolyFromText', 'NumGeometries', 'NumInteriorRings', 'NumPoints', 'Overlaps', 'Point', 'PointFromText', 'PointFromWKB', 'PointN', 'PolygonFromText', 'Polygon', 'SRID', 'StartPoint', 'Touches', 'Within', 'X', 'Y' ); @MYSQL_FUNCTIONS = ( 'AES_DECRYPT', 'AES_ENCRYPT', 'ASYMMETRIC_DECRYPT', 'ASYMMETRIC_DERIVE', 'ASYMMETRIC_ENCRYPT', 'ASYMMETRIC_SIGN', 'ASYMMETRIC_VERIFY', 'CREATE_ASYMMETRIC_PRIV_KEY', 'CREATE_ASYMMETRIC_PUB_KEY', 'CREATE_DH_PARAMETERS', 'CREATE_DIGEST', 'DECODE', 'DES_DECRYPT', 'DES_ENCRYPT', 'ENCODE', 'ENCRYPT', 'SHA1', 'SHA2', 'COLLATION', 'COMPRESS', 'CONVERT', 'DEFAULT', 'FOUND_ROWS', 'GTID_SUBSET', 'GTID_SUBTRACT', 'INET6_ATON', 'INET6_NTOA', 'INTERVAL', 'IS_FREE_LOCK', 'IS_IPV4_COMPAT', 'IS_IPV4_MAPPED', 'IsEmpty', 'LAST_INSERT_ID', 'LOAD_FILE', 'MASTER_POS_WAIT', 'MATCH', 'OLD_PASSWORD', 'PERIOD_ADD', 'PERIOD_DIFF', 'RANDOM_BYTES', 'ROW_COUNT', 'SQL_THREAD_WAIT_AFTER_GTIDS', 'WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS', 'UNCOMPRESS', 'UNCOMPRESSED_LENGTH', 'UpdateXML', 'UUID_SHORT', 'VALIDATE_PASSWORD_STRENGTH', 'WEIGHT_STRING', ); # Scores associated to each code difficulties after replacement. %UNCOVERED_MYSQL_SCORE = ( 'ARRAY_AGG_DISTINCT' => 1, # array_agg(distinct) 'SOUNDS LIKE' => 1, 'CHARACTER SET' => 1, 'COUNT(DISTINCT)' => 2, 'MATCH' => 2, 'JSON' => 2, 'LOCK' => 2, '@VAR' => 0.1, ); @MSSQL_FUNCTIONS = ( 'ATN2', 'CHARINDEX', 'Concat with +', 'CONVERT', 'DATEADD', 'DATEDIFF', 'DATEFROMPARTS', 'DATENAME', 'DATEPART', 'DAY', 'ERROR_LINE', 'ERROR_MESSAGE', 'ERROR_NUMBER', 'ERROR_PROCEDURE', 'ERROR_SEVERITY', 'ERROR_STATE', 'GETDATE', 'GETUTCDATE', 'IIF', 'ISDATE', 'ISNULL', 'ISNUMERIC', 'MONTH', 'NCHAR', 'NULLIF', 'PATINDEX', 'PRINT', 'RAISERROR', 'QUOTENAME', 'RAND', 'REPLICATE', 'SESSIONPROPERTY', 'SOUNDEX', 'SPACE', 'SQUARE', 'STR', 'STUFF', 'SYSDATETIME', 'SYSTEM_USER', 'UNICODE', 'YEAR', ); %UNCOVERED_MSSQL_SCORE = ( 'ARRAY_AGG_DISTINCT' => 1, # array_agg(distinct) 'FOREIGN_OBJECT' => 6, 'SYS_OBJECT' => 6, 'OBJECT_ID' => 2, 'INTO_TEMP_TABLE' => 1, 'GLOBAL_TEMP_TABLE' => 2, 'SELECT_TOP' => 0.2, 'COLLATE' => 0.2, 'RETURNS_TABLE' => 1, 'TODATETIMEOFFSET' => 3, 'CURSOR' => 0.2, 'GLOBAL_VARIABLE' => 1, 'PIVOT' => 12, 'TRY_CATCH' => 3, 'SP_FCT' => 3, 'XML_FCT' => 3, 'FOR_XML' => 6, 'INTERNAL' => 100, ); %EXCEPTION_MAP = ( 'CURSOR_ALREADY_OPEN' => 'duplicate_cursor', 'INVALID_CURSOR' => 'invalid_cursor_state', 'DUP_VAL_ON_INDEX' => 'unique_violation', 'TIMEOUT_ON_RESOURCE' => 'lock_not_available', 'ZERO_DIVIDE' => 'division_by_zero', 'STORAGE_ERROR' => 'out_of_memory', 'INTEGRITY_ERROR' => 'integrity_constraint_violation', 'VALUE_ERROR' => 'data_exception', 'INVALID_NUMBER' => 'data_exception', 'INVALID_CURSOR' => 'invalid_cursor_state', 'NO_DATA_FOUND' => 'no_data_found', 'LOGIN_DENIED' => 'connection_exception', 'TOO_MANY_ROWS'=> 'too_many_rows', 'DUP_VAL_ON_INDEX' => 'unique_violation', 'NOT_LOGGED_ON' => 'insufficient_privilege', 'PROGRAM_ERROR' => 'internal_error', 'ACCESS_INTO_NULL' => 'object_not_in_prerequisite_state', 'COLLECTION_IS_NULL' => 'object_not_in_prerequisite_state', 'INVALID_USERENV_PARAMETER' => 'invalid_parameter_value', # 'ROWTYPE_MISMATCH' => 'DATATYPE MISMATCH' ); %MSSQL_STYLE = ( '100' => 'mon dd yyyy hh:miAM/PM', '101' => 'mm/dd/yyyy', '102' => ' yyyy.mm.dd', '103' => ' dd/mm/yyyy', '104' => 'dd.mm.yyyy', '105' => ' dd-mm-yyyy', '106' => 'dd mon yyyy', '107' => 'Mon dd, yyyy', '108' => 'hh:mm:ss', '109' => 'mon dd yyyy hh:mi:ss:mmmAM (or PM)', '110' => 'mm-dd-yyyy', '111' => ' yyyy/mm/dd', '112' => ' yyyymmdd', '113' => 'dd mon yyyy hh:mi:ss:mmm', '114' => 'hh:mi:ss:mmm', '120' => 'yyyy-mm-dd hh:mi:ss', '121' => 'yyyy-mm-dd hh:mi:ss.mmm', '126' => 'yyyy-mm-ddThh:mi:ss.mmm', '127' => 'yyyy-mm-ddThh:mi:ss.mmmZ', '130' => 'dd mon yyyy hh:mi:ss:mmmAM', '131' => 'dd/mm/yy hh:mi:ss:mmmAM', ); =head1 NAME PSQL - Oracle to PostgreSQL procedural language converter =head1 SYNOPSIS This external perl module is used to convert PLSQL code to PLPGSQL. It is in an external code to allow easy editing and modification. This converter is a work in progress and need your help. It is called internally by Ora2Pg.pm when you set PLSQL_PGSQL configuration option to 1. =cut =head2 convert_plsql_code Main function used to convert Oracle SQL and PL/SQL code into PostgreSQL compatible code =cut sub convert_plsql_code { my ($class, $str, @strings) = @_; return if ($str eq ''); # Remove the SYS schema from calls $str =~ s/\bSYS\.//igs; # Replace outer join sign (+) with a placeholder $class->{outerjoin_idx} //= 0; while ( $str =~ s/\(\+\)/\%OUTERJOIN$class->{outerjoin_idx}\%/s ) { $class->{outerjoin_idx}++; } # Do some initialization of variables %{$class->{single_fct_call}} = (); $class->{replace_out_params} = ''; if (!$self->{use_orafce} && uc($class->{type}) ne 'SHOW_REPORT') { # Rewrite all decode() call before $str = replace_decode($str); # Rewrite numeric operation with ADD_MONTHS(date, 1) to use interval $str =~ s/\b(ADD_MONTHS\s*\([^,]+,\s*\d+\s*\))\s*([+\-\*\/])\s*(\d+)(\s*[^\-\*\/]|$)/$1 $2 '$3 days'::interval$4/sgi; # Rewrite numeric operation with TRUNC() to use interval #$str =~ s/\b(TRUNC\s*\(\s*(?:[^\(\)]+)\s*\))\s*([+\-\*\/])\s*(\d+)(\s*[^+\-\*\/]|$)/$1 $2 '$3 days'::interval$4/sgi; # Rewrite numeric operation with LAST_DAY() to use interval $str =~ s/\b(LAST_DAY\s*\(\s*(?:[^\(\)]+)\s*\))\s*([+\-\*\/])\s*(\d+)(\s*[^+\-\*\/]|$)/$1 $2 '$3 days'::interval$4/sgi; } # Replace array syntax arr(i).x into arr[i].x $str =~ s/\b([a-z0-9_]+)\(([^\(\)]+)\)(\.[a-z0-9_]+)/$1\[$2\]$3/igs; # Extract all block from the code by splitting it on the semi-comma # character and replace all necessary function call my @code_parts = split(/;/, $str); for (my $i = 0; $i <= $#code_parts; $i++) { next if (!$code_parts[$i]); # For mysql also replace if() statements in queries or views. if ($class->{is_mysql} && grep(/^$class->{type}$/i, 'VIEW', 'QUERY', 'FUNCTION', 'PROCEDURE')) { $code_parts[$i] = Ora2Pg::MySQL::replace_if($code_parts[$i]); } # Remove parenthesis from function parameters when they not belong to a function call my %subparams = (); my $p = 0; while ($code_parts[$i] =~ s/(\(\s*)(\([^\(\)]*\))(\s*,)/$1\%SUBPARAMS$p\%$3/is) { $subparams{$p} = $2; $p++; } while ($code_parts[$i] =~ s/(,\s*)(\([^\(\)]*\))(\s*[\),])/$1\%SUBPARAMS$p\%$3/is) { $subparams{$p} = $2; $p++; } # Remove some noisy parenthesis for outer join replacement if ($code_parts[$i] =~ /\%OUTERJOIN\d+\%/) { my %tmp_ph = (); my $idx = 0; while ($code_parts[$i] =~ s/\(([^\(\)]*\%OUTERJOIN\d+\%[^\(\)]*)\)/\%SUBPART$idx\%/s) { $tmp_ph{$idx} = $1; $idx++; } foreach my $k (keys %tmp_ph) { if ($tmp_ph{$k} =~ /^\s*[^\s]+\s*(=|NOT LIKE|LIKE)\s*[^\s]+\s*$/i) { $code_parts[$i] =~ s/\%SUBPART$k\%/$tmp_ph{$k}/s; } else { $code_parts[$i] =~ s/\%SUBPART$k\%/\($tmp_ph{$k}\)/s; } } } %{$class->{single_fct_call}} = (); $code_parts[$i] = extract_function_code($class, $code_parts[$i], 0); # Things that must ne done when functions are replaced with placeholder $code_parts[$i] = replace_without_function($class, $code_parts[$i]); foreach my $k (keys %{$class->{single_fct_call}}) { $class->{single_fct_call}{$k} = replace_oracle_function($class, $class->{single_fct_call}{$k}, @strings); if ($class->{single_fct_call}{$k} =~ /^CAST\s*\(/i) { $class->{single_fct_call}{$k} = replace_sql_type($class, $class->{single_fct_call}{$k}); } if ($class->{single_fct_call}{$k} =~ /^CAST\s*\(.*\%\%REPLACEFCT(\d+)\%\%/i) { $class->{single_fct_call}{$1} = replace_sql_type($class, $class->{single_fct_call}{$1}); } } while ($code_parts[$i] =~ s/\%\%REPLACEFCT(\d+)\%\%/$class->{single_fct_call}{$1}/) {}; $code_parts[$i] =~ s/\%SUBPARAMS(\d+)\%/$subparams{$1}/igs; # Remove potential double affectation for function with out parameter $code_parts[$i] =~ s/(\s*)[^\s=;]+\s*:=\s*(?:\%ORA2PG_COMMENT\d+\%)?(\s*[^\s;=]+\s*:=)/$1$2/gs; $code_parts[$i] =~ s/(\s*)[^\s=;]+\s*:=\s*(SELECT\s+[^;]+INTO\s*)/$1$2/igs; } $str = join(';', @code_parts); if ($class->{replace_out_params}) { if ($str !~ s/\b(DECLARE(?:\s+|\%ORA2PG_COMMENT\d+\%))/$1$class->{replace_out_params}\n/is) { $str =~ s/\b(BEGIN(?:\s+|\%ORA2PG_COMMENT\d+\%))/DECLARE\n$class->{replace_out_params}\n$1/is; } $class->{replace_out_params} = ''; } # Apply code rewrite on other part of the code $str = plsql_to_plpgsql($class, $str, @strings); if ($class->{get_diagnostics}) { if ($str !~ s/\b(DECLARE\s+)/$1$class->{get_diagnostics}\n/is) { $str =~ s/\b(BEGIN\s+)/DECLARE\n$class->{get_diagnostics}\n$1/is; } $class->{get_diagnostics} = ''; } return $str; } sub clear_parenthesis { my $str = shift; # Keep parenthesys with sub queries if ($str =~ /\bSELECT\b/i) { $str = '((' . $str . '))'; } else { $str =~ s/^\s+//s; $str =~ s/\s+$//s; $str = '(' . $str . ')'; } return $str; } =head2 extract_function_code Recursive function used to extract call to function in Oracle SQL and PL/SQL code =cut sub extract_function_code { my ($class, $code, $idx) = @_; # Remove some extra parenthesis for better parsing $code =~ s/\(\s*\(([^\(\)]*)\)\s*\)/clear_parenthesis($1)/iges; # Look for a function call that do not have an other function # call inside, replace content with a marker and store the # replaced string into a hask to rewritten later to convert pl/sql if ($code =~ s/\b([a-zA-Z0-9\.\_]+)\s*\(([^\(\)]*)\)/\%\%REPLACEFCT$idx\%\%/s) { my $fct_name = $1; my $fct_code = $2; my $space = ''; $space = ' ' if (grep (/^$fct_name$/i, 'FROM', 'AS', 'VALUES', 'DEFAULT', 'OR', 'AND', 'IN', 'SELECT', 'OVER', 'WHERE', 'THEN', 'IF', 'ELSIF', 'ELSE', 'EXISTS', 'ON')); # Move up any outer join inside a function otherwise it will not be detected my $outerjoin = ''; if ($fct_code =~ /\%OUTERJOIN(\d+)\%/s) { my $idx_join = $1; # only if the placeholder content is a function not a predicate if ($fct_code !~ /(=|>|<|LIKE|NULL|BETWEEN)/i) { $fct_code =~ s/\%OUTERJOIN$idx_join\%//s; $outerjoin = "\%OUTERJOIN$idx_join\%"; } } # recursively replace function $class->{single_fct_call}{$idx} = $fct_name . $space . '(' . $fct_code . ')' . $outerjoin; $code = extract_function_code($class, $code, ++$idx); } return $code; } sub append_alias_clause { my $str = shift; # Divise code through UNION keyword marking a new query level my @q = split(/\b(UNION\s+ALL|UNION|EXCEPT)\b/i, $str); for (my $j = 0; $j <= $#q; $j+=2) { if ($q[$j] =~ s/\b(FROM\s+)(.*\%SUBQUERY.*?)(\s*)(WHERE|ORDER\s+BY|GROUP\s+BY|LIMIT|$)/$1\%FROM_CLAUSE\%$3$4/is) { my $from_clause = $2; if ($from_clause !~ /TABLE\%SUBQUERY\d+\%/ && $q[$j] !~ /\b(YEAR|MONTH|DAY|HOUR|MINUTE|SECOND|TIMEZONE_HOUR|TIMEZONE_MINUTE|TIMEZONE_ABBR|TIMEZONE_REGION|TIMEZONE_OFFSET)\s+FROM/is) { my @parts = split(/\b(WHERE|ORDER\s+BY|GROUP\s+BY|LIMIT)\b/i, $from_clause); $parts[0] =~ s/(?=]|NOT LIKE|LIKE|WHERE|GROUP|ORDER)/is) { $str =~ s/(END\b\s*)[\w"\.]+\s*(?:;|$)/$1;/is; } return $str; } =head2 set_error_code Transform custom exception code by replacing the leading -20 by 45 =cut sub set_error_code { my $code = shift; my $orig_code = $code; $code =~ s/-20(\d{3})/'45$1'/; if ($code =~ s/-20(\d{2})/'450$1'/ || $code =~ s/-20(\d{1})/'4500$1'/) { print STDERR "WARNING: exception code has less than 5 digit, proceeding to automatic adjustement.\n"; $code .= " /* code was: $orig_code */"; } return $code; } # Fix case where the raise_application_error() parameters are named by removing them sub remove_named_parameters { my $str = shift; $str =~ s/\w+\s*=>\s*//g; return $str; } sub set_interval_value { my $num = shift(); if ($num !~ /\./) { return "'$num days'"; } else { return "'" . int($num*86400) . " seconds'"; } } =head2 plsql_to_plpgsql This function return a PLSQL code translated to PLPGSQL code =cut sub plsql_to_plpgsql { my ($class, $str, @strings) = @_; return if ($str eq ''); return mysql_to_plpgsql($class, $str, @strings) if ($class->{is_mysql}); return mssql_to_plpgsql($class, $str, @strings) if ($class->{is_mssql}); my $field = '\s*([^\(\),]+)\s*'; my $num_field = '\s*([\d\.]+)\s*'; # Rewrite variable calls $str =~ s/([a-z=<>\*\+\-\/\( ]): ([a-z0-9_\$]+)/$1 :'$2'/igs if ($class->{type} eq 'QUERY'); my $conv_current_time = 'clock_timestamp()'; if (!grep(/$class->{type}/i, 'FUNCTION', 'PROCEDURE', 'PACKAGE')) { $conv_current_time = 'statement_timestamp()'; } # Remove the SYS schema from calls $str =~ s/\bSYS\.//igs; # Replace sysdate +/- N by localtimestamp - N day interval $str =~ s/\bSYSDATE\s*(\+|\-)\s*([\d\.]+)/"$conv_current_time $1 interval " . set_interval_value($2)/iges; # Replace special case : (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 # with: (extract(epoch from now()) # When translating from code while ($str =~ /\bSYSDATE\s*\-\s*to_date\(\s*\?TEXTVALUE(\d+)\?\s*,\s*\?TEXTVALUE(\d+)\?\s*\)\s*\)(?:\s*\*\s*(?:24|60)){3}/is) { my $t1 = $1; my $t2 = $2; if ($class->{text_values}{$t1} =~ /^'(Jan|01|1970|\.|\-)+'$/ && $class->{text_values}{$t2} =~ /'(Mon|mm|dd|yyyy|\.|\-)+'/i) { $str =~ s/\bSYSDATE\s*\-\s*to_date\(\s*\?TEXTVALUE(\d+)\?\s*,\s*\?TEXTVALUE(\d+)\?\s*\)\s*\)(?:\s*\*\s*(?:24|60)){3}/extract(epoch from now()))/is; } } # When translating from default value (sysdate - to_date('01-01-1970','dd-MM-yyyy'))*24*60*60 $str =~ s/\bSYSDATE\s*\-\s*to_date\(\s*'(Jan|01|1970|\.|\-)+'\s*,\s*'(Mon|mm|dd|yyyy|\.|\-)+'\s*\)\s*\)(\s*\*\s*(24|60)){3}/extract(epoch from now()))/igs; # Change SYSDATE to 'now' or current timestamp. $str =~ s/\bSYSDATE\s*\(\s*\)/$conv_current_time/igs; $str =~ s/\bSYSDATE\b/$conv_current_time/igs; # Cast call to to_date with localtimestamp $str =~ s/(TO_DATE\($conv_current_time)\s*,/$1::text,/igs; # JSON validation mostly in CHECK contraints $str =~ s/((?:\w+\.)?\w+)\s+IS\s+JSON\b/\(CASE WHEN $1::json IS NULL THEN true ELSE true END\)/igs; # replace the modulo operator $str =~ s/([^\(\s]+)\s+MOD\s+([^\s\)]+)/mod\($1, $2\)/igs; # Drop temporary doesn't exist in PostgreSQL $str =~ s/DROP\s+TEMPORARY/DROP/igs; # Private temporary table doesn't exist in PostgreSQL $str =~ s/PRIVATE\s+TEMPORARY/TEMPORARY/igs; $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; # Replace SYSTIMESTAMP $str =~ s/\bSYSTIMESTAMP\b/statement_timestamp()/igs; # remove FROM DUAL $str =~ s/FROM\s+DUAL//igs; $str =~ s/FROM\s+SYS\.DUAL//igs; # DISTINCT and UNIQUE are synonym on Oracle $str =~ s/SELECT\s+UNIQUE\s+([^,])/SELECT DISTINCT $1/igs; # Remove space between operators $str =~ s/=\s+>/=>/gs; $str =~ s/<\s+=/<=/gs; $str =~ s/>\s+=/>=/gs; $str =~ s/!\s+=/!=/gs; $str =~ s/<\s+>/<>/gs; $str =~ s/:\s+=/:=/gs; $str =~ s/\|\s+\|/\|\|/gs; $str =~ s/!=([+\-])/!= $1/gs; # replace operator for named parameters in function calls if (!$class->{pg_supports_named_operator}) { $str =~ s/([^<])=>/$1:=/gs; } # replace the BITAND function by the & operator $str =~ s/BITAND\(\s*([^\(,]+)\s*,\s*([^\)]+)\s*\)/($1 & $2)/igs; # Replace listagg() call $str =~ s/\bLISTAGG\s*\((.*?)(?:\s*ON OVERFLOW [^\)]+)?\)\s*WITHIN\s+GROUP\s*\((.*?)\)/string_agg($1 $2)/igs; # Try to fix call to string_agg with a single argument (allowed in oracle) $str =~ s/\bstring_agg\(([^,\(\)]+)\s+(ORDER\s+BY)/string_agg($1, '' $2/igs; # There's no such things in PostgreSQL $str =~ s/PRAGMA RESTRICT_REFERENCES[^;]+;//igs; $str =~ s/PRAGMA SERIALLY_REUSABLE[^;]*;//igs; $str =~ s/PRAGMA INLINE[^;]+;//igs; # There are no autonomous transactions in standard postgres (as of version 15) my $unsupported = ''; $unsupported = "-- Unsupported, consider using dblink to emulate oracle behavior or see AUTONOMOUS_TRANSACTION in ora2pg.conf" if (!$self->{autonomous_transaction}); $str =~ s/[ ]+PRAGMA\s+AUTONOMOUS_TRANSACTION;/$unsupported\n-- $&/igs; # Remove the extra TRUNCATE clauses not available in PostgreSQL $str =~ s/TRUNCATE\s+TABLE\s+(.*?)\s+(REUSE|DROP)\s+STORAGE/TRUNCATE TABLE $1/igs; $str =~ s/TRUNCATE\s+TABLE\s+(.*?)\s+(PRESERVE|PURGE)\s+MATERIALIZED\s+VIEW\s+LOG/TRUNCATE TABLE $1/igs; # Converting triggers # :new. -> NEW. $str =~ s/:new\./NEW\./igs; # :old. -> OLD. $str =~ s/:old\./OLD\./igs; # Change NVL to COALESCE $str =~ s/NVL\s*\(/coalesce(/isg; $str =~ s/NVL2\s*\($field,$field,$field\)/(CASE WHEN $1 IS NOT NULL THEN $2 ELSE $3 END)/isg; # NLSSORT to COLLATE while ($str =~ /NLSSORT\($field,$field[\)]?/is) { my $col = $1; my $nls_sort = $2; if ($nls_sort =~ s/\%\%string(\d+)\%\%/$strings[$1]/s) { $nls_sort =~ s/NLS_SORT=([^']+).*/COLLATE "$1"/is; $nls_sort =~ s/\%\%ESCAPED_STRING\%\%//ig; $str =~ s/NLSSORT\($field,$field[\)]?/$1 $nls_sort/is; } elsif ($nls_sort =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/s) { $nls_sort =~ s/\s*'NLS_SORT=([^']+).*/COLLATE "$1"/is; $nls_sort =~ s/\%\%ESCAPED_STRING\%\%//ig; $str =~ s/NLSSORT\($field,$field[\)]?/$1 $nls_sort/is; } else { $str =~ s/NLSSORT\($field,\s*'NLS_SORT=([^']+)'\s*[\)]?/$1 COLLATE "$2"/is; $str =~ s/NLSSORT\($field,$field[\)]?/$1 COLLATE $2/is; } } # Replace EXEC function into variable, ex: EXEC :a := test(:r,1,2,3); $str =~ s/\bEXEC\s+:([^\s:]+)\s*:=/SELECT INTO $2/igs; # Replace simple EXEC function call by SELECT function $str =~ s/\bEXEC(\s+)(?:.*?)(?!FROM\b|WHERE\b)/SELECT$1/igs; # Remove leading : on Oracle variable taking care of regex character class $str =~ s/([^\w:]+):(\d+)/$1\$$2/igs; $str =~ s/([^\w:]+):((?!alpha:|alnum:|blank:|cntrl:|digit:|graph:|lower:|print:|punct:|space:|upper:|xdigit:)\w+)/$1$2/igs; # INSERTING|DELETING|UPDATING -> TG_OP = 'INSERT'|'DELETE'|'UPDATE' $str =~ s/\bINSERTING\b/TG_OP = 'INSERT'/igs; $str =~ s/\bDELETING\b/TG_OP = 'DELETE'/igs; $str =~ s/\bUPDATING\b/TG_OP = 'UPDATE'/igs; # Replace Oracle call to column in trigger event $str =~ s/TG_OP = '([^']+)'\s*\(\s*([^\)]+)\s*\)/TG_OP = '$1' AND NEW.$2 IS DISTINCT FROM OLD.$2/igs; # EXECUTE IMMEDIATE => EXECUTE $str =~ s/EXECUTE IMMEDIATE/EXECUTE/igs; # DBMS_SQL.open_cursor, DBMS_SQL.parse, DBMS_SQL.execute, DBMS_SQL.close_cursor => EXECUTE / GET DIAGNOSTICS = ROW COUNT $str =~ s/([ \t]*)([^\:\s]+)\s*:=\s*DBMS_SQL\.open_cursor\s*;(.*?)DBMS_SQL\.parse\s*\([^\),]+,\s*([^\),]+)(?:\s*,[^\)]+)?\)\s*;(?:\%ORA2PG_COMMENT\d+\%|\s)*([^\:\s]+)\s*:=\s*DBMS_SQL\.execute\([^\)]+\);(.*?)DBMS_SQL\.close_cursor\s*\([^\),]+\);/$3EXECUTE $4;\n$1GET DIAGNOSTICS $5 = ROW_COUNT;/igs; $str =~ s/([\t ]*)([^\:\s]+)\s*:=\s*DBMS_SQL\.open_cursor\s*;(.*?)DBMS_SQL\.parse\s*\([^\),]+,\s*([^\),]+)(?:\s*,[^\)]+)?\)\s*;(?:\%ORA2PG_COMMENT\d+\%|\s)*DBMS_SQL\.execute\([^\)]+\);(.*?)DBMS_SQL\.close_cursor\s*\([^\),]+\);/$3EXECUTE $4;/igs; # SELECT without INTO should be PERFORM. Exclude select of view when prefixed with AS ot IS if ( !grep(/^$class->{type}$/, 'QUERY', 'VIEW', 'SCRIPT') ) { $str =~ s/(\s+)(?{export_schema}) { if (!$class->{preserve_case}) { $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('\L$2\E')/isg; $str =~ s/\b(\w+)\.(\w+)\.currval/currval('\L$2\E')/isg; } else { $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('"$2"')/isg; $str =~ s/\b(\w+)\.(\w+)\.currval/currval('"$2"')/isg; } } else { my $sch = $class->{pg_schema} || $class->{schema}; if (!$class->{preserve_case}) { $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('\L$sch.$2\E')/isg; $str =~ s/\b(\w+)\.(\w+)\.currval/currval('\L$sch.$2\E')/isg; } else { $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('"$sch"."$2"')/isg; $str =~ s/\b(\w+)\.(\w+)\.currval/currval('"$sch"."$2"')/isg; } } if (!$class->{preserve_case}) { $str =~ s/\b(\w+)\.nextval/nextval('\L$1\E')/isg; $str =~ s/\b(\w+)\.currval/currval('\L$1\E')/isg; } else { $str =~ s/\b(\w+)\.nextval/nextval('"$1"')/isg; $str =~ s/\b(\w+)\.currval/currval('"$1"')/isg; } # Oracle MINUS can be replaced by EXCEPT as is $str =~ s/\bMINUS\b/EXCEPT/igs; # Comment DBMS_OUTPUT.ENABLE calls if (!$class->{use_orafce}) { $str =~ s/(DBMS_OUTPUT.ENABLE[^;]+;)/-- $1/isg; } # DBMS_LOB.GETLENGTH can be replaced by binary length. $str =~ s/DBMS_LOB.GETLENGTH/octet_length/igs; # DBMS_LOB.SUBSTR can be replaced by SUBSTR() $str =~ s/DBMS_LOB.SUBSTR\s*\($field,$field,$field\)/substr($1, $3, $2)/igs; # TO_CLOB(), we just remove it $str =~ s/TO_CLOB\s*\(/\(/igs; # Raise information to the client if (!$class->{use_orafce}) { $str =~ s/DBMS_OUTPUT\.(put_line|put|new_line)\s*\((.*?)\)\s*;/&raise_output($class, $2) . ';'/isge; } else { $str =~ s/(DBMS_OUTPUT\.)/PERFORM $1/igs; } # DBMS_LOCK.SLEEP can be replaced by pg_sleep $str =~ s/DBMS_(LOCK|SESSION)\.SLEEP/PERFORM pg_sleep/igs; # Simply remove this as not supported $str =~ s/\bDEFAULT\s+NULL\b//igs; # Fix some reserved keyword that could be used in a query $str =~ s/(\s+)(month|year)([\s,])/$1"$2"$3/igs; # Replace DEFAULT empty_blob() and empty_clob() my $empty = "''"; $empty = 'NULL' if ($class->{empty_lob_null}); $str =~ s/(empty_blob|empty_clob)\s*\(\s*\)/$empty/is; $str =~ s/(empty_blob|empty_clob)\b/$empty/is; # dup_val_on_index => unique_violation : already exist exception $str =~ s/\bdup_val_on_index\b/unique_violation/igs; # Replace raise_application_error by PG standard RAISE EXCEPTION $str =~ s/\braise_application_error\s*\(\s*([^,]+)\s*,\s*([^;]+),\s*(true|false)\s*\)\s*;/"RAISE EXCEPTION '%', " . remove_named_parameters($2) . " USING ERRCODE = " . set_error_code(remove_named_parameters($1)) . ";"/iges; $str =~ s/\braise_application_error\s*\(\s*([^,]+)\s*,\s*([^;]+)\)\s*;/"RAISE EXCEPTION '%', " . remove_named_parameters($2) . " USING ERRCODE = " . set_error_code(remove_named_parameters($1)) . ";"/iges; $str =~ s/DBMS_STANDARD\.RAISE EXCEPTION/RAISE EXCEPTION/igs; # Translate cursor declaration $str = replace_cursor_def($str); # Remove remaining %ROWTYPE in other prototype declaration #$str =~ s/\%ROWTYPE//isg; # Normalize HAVING ... GROUP BY into GROUP BY ... HAVING clause # $str =~ s/\bHAVING\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM).)*?)\bGROUP BY\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM).)*?)((?=UNION|ORDER BY|LIMIT|INTO |FOR UPDATE|PROCEDURE|\)\s+(?:AS)*[a-z0-9_]+\s+)|$)/GROUP BY$2 HAVING$1/gis; # Modified Regex that can hanlle group by and having when having clause inside subquery $str =~ s/\bHAVING\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM|\().)*?)\bGROUP BY\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM|\().)*?)((?=UNION|ORDER BY|LIMIT|INTO |FOR UPDATE|PROCEDURE|\)\s+(?:AS)*[a-z0-9_]+\s+)|$)/GROUP BY$2 HAVING$1/gis; # Add STRICT keyword when select...into and an exception with NO_DATA_FOUND/TOO_MANY_ROW is present #$str =~ s/\b(SELECT\b[^;]*?INTO)(.*?)(EXCEPTION.*?(?:NO_DATA_FOUND|TOO_MANY_ROW))/$1 STRICT $2 $3/igs; # Add STRICT keyword when SELECT...INTO or EXECUTE ... INTO even if there's not EXCEPTION block $str =~ s/\b((?:SELECT|EXECUTE)\s+[^;]*?\s+INTO)(\s+(?!STRICT))/$1 STRICT$2/igs; $str =~ s/(INSERT\s+INTO\s+)STRICT\s+/$1/igs; # Remove the function name repetion at end $str =~ s/\b(END\s*[^;\s]+\s*(?:;|$))/remove_fct_name($1)/iges; # Rewrite comment in CASE between WHEN and THEN $str =~ s/(\s*)(WHEN\s+[^\s]+\s*)(\%ORA2PG_COMMENT\d+\%)(\s*THEN)/$1$3$1$2$4/igs; # Replace SQLCODE by SQLSTATE $str =~ s/\bSQLCODE\b/SQLSTATE/igs; # Revert order in FOR IN REVERSE $str =~ s/\bFOR(.*?)IN\s+REVERSE\s+([^\.\s]+)\s*\.\.\s*([^\s]+)/FOR$1IN REVERSE $3..$2/isg; # Comment call to COMMIT or ROLLBACK in the code if allowed if ($class->{comment_commit_rollback}) { $str =~ s/\b(COMMIT|ROLLBACK)\s*;/-- $1;/igs; $str =~ s/(ROLLBACK\s+TO\s+[^;]+);/-- $1;/igs; } # Comment call to SAVEPOINT in the code if allowed if ($class->{comment_savepoint}) { $str =~ s/(SAVEPOINT\s+[^;]+);/-- $1;/igs; } # Replace exit at end of cursor $str =~ s/EXIT\s+WHEN\s+([^\%;]+)\%\s*NOTFOUND\s*;/EXIT WHEN NOT FOUND; \/\* apply on $1 \*\//isg; $str =~ s/EXIT\s+WHEN\s+\(\s*([^\%;]+)\%\s*NOTFOUND\s*\)\s*;/EXIT WHEN NOT FOUND; \/\* apply on $1 \*\//isg; # Same but with additional conditions $str =~ s/EXIT\s+WHEN\s+([^\%;]+)\%\s*NOTFOUND\s+([^;]+);/EXIT WHEN NOT FOUND $2; \/\* apply on $1 \*\//isg; $str =~ s/EXIT\s+WHEN\s+\(\s*([^\%;]+)\%\s*NOTFOUND\s+([^\)]+)\)\s*;/EXIT WHEN NOT FOUND $2; \/\* apply on $1 \*\//isg; # Replacle call to SQL%NOTFOUND and SQL%FOUND $str =~ s/SQL\s*\%\s*NOTFOUND/NOT FOUND/isg; $str =~ s/SQL\s*\%\s*FOUND/FOUND/isg; # Replace all remaining CURSORNAME%NOTFOUND with NOT FOUND $str =~ s/\s+([^\(\%\s]+)\%\s*NOTFOUND\s*/ NOT FOUND /isg; # Replace UTL_MATH function by fuzzymatch function $str =~ s/UTL_MATCH.EDIT_DISTANCE/levenshtein/igs; # Replace UTL_ROW.CAST_TO_RAW function by encode function $str =~ s/UTL_RAW.CAST_TO_RAW\s*\(\s*([^\)]+)\s*\)/encode($1::bytea, 'hex')::bytea/igs; # Replace known EXCEPTION equivalent ERROR code foreach my $e (keys %EXCEPTION_MAP) { $str =~ s/\b$e\b/$EXCEPTION_MAP{"\U$e\L"}/igs; } # Replace special IEEE 754 values for not a number and infinity $str =~ s/BINARY_(FLOAT|DOUBLE)_NAN/'NaN'/igs; $str =~ s/([\-]*)BINARY_(FLOAT|DOUBLE)_INFINITY/'$1Infinity'/igs; $str =~ s/'([\-]*)Inf'/'$1Infinity'/igs; # Replace PIPE ROW by RETURN NEXT $str =~ s/PIPE\s+ROW\s*/RETURN NEXT /igs; $str =~ s/(RETURN NEXT )\(([^\)]+)\)/$1$2/igs; # Convert all x <> NULL or x != NULL clauses to x IS NOT NULL. $str =~ s/\s*(<>|\!=)\s*NULL/ IS NOT NULL/igs; # Convert all x = NULL clauses to x IS NULL. $str =~ s/(?!:)(.)=\s*NULL/$1 IS NULL/igs; # Add missing FROM clause in DELETE statements minus MERGE and FK ON DELETE $str =~ s/(\bDELETE\s+)(?!FROM|WHERE|RESTRICT|CASCADE|NO ACTION)\b/$1FROM /igs; # Revert changes on update queries for IS NULL transaltion in the target list only while ($str =~ s/\b(UPDATE\s+((?!WHERE|;).)*)\s+IS NULL/$1 = NULL/is) {}; # Rewrite all IF ... IS NULL with coalesce because for Oracle empty and NULL is the same if ($class->{null_equal_empty}) { # Form: column IS NULL $str =~ s/([a-z0-9_\."]+)\s*IS\s+NULL/coalesce($1::text, '') = ''/igs; my $i = 0; my %isnull = (); while ($str =~ s/([a-z0-9_\."]+)\s*IS\s+NULL/%ORA2PGISNULL$i%/is) { $isnull{$i} = "coalesce($1::text, '') = ''"; $i++; } my %notnull = (); while ($str =~ s/([a-z0-9_\."]+)\s*IS NOT NULL/%ORA2PGNOTNULL$i%/is) { $notnull{$i} = "($1 IS NOT NULL AND $1::text <> '')"; $i++; } # Form: fct(expression) IS NULL $str =~ s/([a-z0-9_\."]+\s*\([^\)\(]*\))\s*IS\s+NULL/coalesce($1::text, '') = ''/igs; $str =~ s/([a-z0-9_\."]+\s*\([^\)\(]*\))\s*IS\s+NOT\s+NULL/($1 IS NOT NULL AND ($1)::text <> '')/igs; $str =~ s/%ORA2PGISNULL(\d+)%/$isnull{$1}/gs; $str =~ s/%ORA2PGNOTNULL(\d+)%/$notnull{$1}/gs; } # Replace type in sub block $str =~ s/(BEGIN.*?DECLARE\s+)(.*?)(\s+BEGIN)/$1 . replace_sql_type($class, $2) . $3/iges; # Replace type in RETURNING clauses $str =~ s/(RETURNING\s+)([^\(\)]+(?:\(\d+\))?)/$1 . replace_sql_type($class, $2)/iges; # Remove any call to MDSYS schema in the code $str =~ s/\bMDSYS\.//igs; # Oracle doesn't require parenthesis after VALUES, PostgreSQL has # similar proprietary syntax but parenthesis are mandatory $str =~ s/(INSERT\s+INTO\s+(?:.*?)\s+VALUES\s+)([^\(\)\s]+)\s*;/$1\($2.*\);/igs; # Replace some windows function issues with KEEP (DENSE_RANK FIRST ORDER BY ...) $str =~ s/\b(MIN|MAX|SUM|AVG|COUNT|VARIANCE|STDDEV)\s*\(([^\)]+)\)\s+KEEP\s*\(DENSE_RANK\s+(FIRST|LAST)\s+(ORDER\s+BY\s+[^\)]+)\)\s*(OVER\s*\(PARTITION\s+BY\s+[^\)]+)\)/$3_VALUE($2) $5 $4)/igs; $class->{sub_queries} = (); $class->{sub_queries_idx} = 0; #### # Replace ending ROWNUM with LIMIT or row_number() and replace (+) outer join #### # Catch potential subquery first and replace rownum in subqueries my @statements = split(/;/, $str); for ( my $i = 0; $i <= $#statements; $i++ ) { # Remove any unecessary parenthesis in code $statements[$i] = remove_extra_parenthesis($statements[$i]); $class->{sub_parts} = (); $class->{sub_parts_idx} = 0; extract_subpart($class, \$statements[$i]); # Translate all sub parts of the query before applying translation on the main query foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) { if ($class->{sub_parts}{$z} =~ /\S/is) { $class->{sub_parts}{$z} = translate_statement($class, $class->{sub_parts}{$z}, 1); if ($class->{sub_parts}{$z} =~ /SELECT/is) { $class->{sub_parts}{$z} .= $class->{limit_clause}; $class->{limit_clause} = ''; } # Try to append aliases of subqueries in the from clause $class->{sub_parts}{$z} = append_alias_clause($class->{sub_parts}{$z}); } next if ($class->{sub_parts}{$z} =~ /^\(/ || $class->{sub_parts}{$z} =~ /^TABLE[\(\%]/i); # If subpart is not empty after transformation if ($class->{sub_parts}{$z} =~ /\S/is) { # add open and closed parenthesis $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) { # otherwise do not report the empty parenthesis when this is not a function $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } } # Try to append aliases of subqueries in the from clause $statements[$i] = append_alias_clause($statements[$i]); $statements[$i] .= $class->{limit_clause}; $class->{limit_clause} = ''; # Apply translation on the full query $statements[$i] = translate_statement($class, $statements[$i]); $statements[$i] .= $class->{limit_clause}; $class->{limit_clause} = ''; # then restore subqueries code into the main query while ($statements[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) {}; # Remove unnecessary offset to position 0 which is the default $statements[$i] =~ s/\s+OFFSET 0//igs; } map { s/[ ]+([\r\n]+)/$1/s; } @statements; map { s/[ ]+$//; } @statements; $str = join(';', @statements); # Rewrite some garbadged resulting from the transformation while ($str =~ s/(\s+AND)\s+AND\b/$1/is) {}; while ($str =~ s/(\s+OR)\s+OR\b/$1/is) {}; while ($str =~ s/\s+AND(\s+\%ORA2PG_COMMENT\d+\%\s+)+(AND)\b/$1$2/is) {}; while ($str =~ s/\s+OR(\s+\%ORA2PG_COMMENT\d+\%\s+)+(OR)\b/$1$2/is) {}; $str =~ s/\(\s*(AND|OR)\b/\(/igs; $str =~ s/(\s+WHERE)\s+(AND|OR)\b/$1/igs; $str =~ s/(\s+WHERE)(\s+\%ORA2PG_COMMENT\d+\%\s+)+(AND|OR)\b/$1$2/igs; # Attempt to remove some extra parenthesis in simple case only $str = remove_extra_parenthesis($str); # Remove cast in partition range $str =~ s/TIMESTAMP\s*('[^']+')/$1/igs; # Replace call to SQL%ROWCOUNT $str =~ s/([^\s:]+)\s*:=\s*SQL\%ROWCOUNT/GET DIAGNOSTICS $1 = ROW_COUNT/igs; if ($str =~ s/(IF\s+)[^\%\s;]+\%ROWCOUNT/GET DIAGNOSTICS ora2pg_rowcount = ROW_COUNT;\n$1ora2pg_rowcount/igs) { $class->{get_diagnostics} = 'ora2pg_rowcount int;'; } elsif ($str =~ s/;(\s+)([^;]+)SQL\%ROWCOUNT/;$1GET DIAGNOSTICS ora2pg_rowcount = ROW_COUNT;\n$1$2 ora2pg_rowcount/igs) { $class->{get_diagnostics} = 'ora2pg_rowcount int;'; } # SQL%ROWCOUNT with concatenated string $str =~ s/(\s+)(GET DIAGNOSTICS )([^\s]+)( = ROW_COUNT)(\s+\|\|[^;]+);/$1$2$3$4;$1$3 := $3 $5;/; # Replace call of ROWNUM in the target list with row_number() over () $str =~ s/(PERFORM|SELECT|,|\()\s*ROWNUM\b((?:.*?)\s+FROM\s+)/$1row_number() over ()$2/igs; # Sometime variable used in FOR ... IN SELECT loop is not declared # Append its RECORD declaration in the DECLARE section. my $tmp_code = $str; while ($tmp_code =~ s/\bFOR\s+([^\s]+)\s+IN(.*?)LOOP//is) { my $varname = $1; my $clause = $2; my @code = split(/\bBEGIN\b/i, $str); if ($code[0] !~ /\bDECLARE\s+.*\b$varname\s+/is) { # When the cursor is refereing to a statement, declare # it as record otherwise it don't need to be replaced if ($clause =~ /\bSELECT\b/is) { # append variable declaration to declare section if ($str !~ s/\bDECLARE\b/DECLARE\n $varname RECORD;/is) { # No declare section $str = "DECLARE\n $varname RECORD;\n" . $str; } } } } # Rewrite direct call to function without out parameters using PERFORM $str = perform_replacement($class, $str); # Restore non converted outer join $str =~ s/\%OUTERJOIN\d+\%/\(\+\)/igs; # Rewrite some SQL script setting from Oracle $str =~ s/[\r\n]set\s+timing\s+(on|off)/\n\\timing $1/igs; $str =~ s/[\r\n]set\s+heading\s+on/\n\\pset tuples_only off/igs; $str =~ s/[\r\n]set\s+heading\s+off/\n\\pset tuples_only on/igs; $str =~ s/[\r\n]set\s+heading\s+(on|off)/\n\\pset tuple_only $1/igs; $str =~ s/[\r\n](set\s+serveroutput\s+.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+showmod\s+.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+verify\s+.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+trimspool\s+.*)/\n--$1/igs; $str =~ s/[\r\n](rem\s+.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+(?:array|arraysize)\s+\d+)/\n-- $1/igs; $str =~ s/[\r\n]set\s+(?:auto|autocommit)\s+(on|off)/\n\\set AUTOCOMMIT $1/igs; $str =~ s/[\r\n]set\s+echo\s+on/\n\\set ECHO queries/igs; $str =~ s/[\r\n]set\s+echo\s+off/\n\\set ECHO none/igs; $str =~ s/[\r\n]set\s+(?:heading|head)\s+(on|off)/\n\\pset tuples_only $1/igs; $str =~ s/[\r\n]set\s+(?:trim|trimout)\s+on/\n\\pset format unaligned/igs; $str =~ s/[\r\n]set\s+(?:trim|trimout)\s+off/\n\\pset format aligned/igs; $str =~ s/[\r\n]set\s+colsep\s+([^\s]+)/\n\\pset fieldsep $1/igs; $str =~ s/[\r\n]spool\s+off/\n\\o/igs; $str =~ s/[\r\n]spool\s+([^\s]*)/\n\\o $1/igs; $str =~ s/[\r\n]ttitle\s+/\n\\pset title /igs; $str =~ s/[\r\n]prompt\s+/\n\\qecho /igs; $str =~ s/[\r\n]set\s+feedback\s+off/\n\\set QUIET on/igs; $str =~ s/[\r\n]set\s+pagesize\s+0/\n\\pset pager off/igs; $str =~ s/[\r\n](set\s+pagesize\s+\d+)/\n--$1/igs; $str =~ s/[\r\n]set\s+linesize\s+0/\n\\pset pager off/igs; $str =~ s/[\r\n](set\s+linesize\s+\d+)/\n--$1/igs; $str =~ s/[\r\n](set\s+termout.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+newpage.*)/\n--$1/igs; $str =~ s/[\r\n](set\s+(?:linesize|pagesize|feedback|verify)\s+)/\n--$1/igs; $str =~ s/[\r\n](disconnect)\b/\n--$1/igs; $str =~ s/[\r\n](connect\s+)/\n--$1/igs; $str =~ s/[\r\n](quit)(\s*;)?\b/\n\\$1/igs; $str =~ s/[\r\n]!/\n\\! /gs; $str = replace_sql_type($class, $str); #column c1 format a8 head 'date my @content = split(/[\n]/, $str); my %format_col_title = (); for (my $i = 0; $i <= $#content; $i++) { if ($content[$i] =~ s/^(column\s+([^\s]+)\s+.*head (\?TEXTVALUE\d+\?))/--$1/is) { $format_col_title{$2} = $3; } else { foreach my $k (keys %format_col_title) { if ($format_col_title{$k} =~ /\?TEXTVALUE(\d+)\?/) { $class->{text_values}{$1} =~ s/["']//g; } if ($content[$i] !~ /\bo[nf]\s+$k\b/) { $content[$i] =~ s/\b$k\b/$k as "$format_col_title{$k}"/i; } } } } $str = join("\n", @content); return $str; } ############## # Rewrite direct call to function without out parameters using PERFORM ############## sub perform_replacement { my ($class, $str) = @_; if (uc($class->{type}) =~ /^(PACKAGE|FUNCTION|PROCEDURE|TRIGGER)$/) { foreach my $sch ( keys %{ $class->{function_metadata} }) { foreach my $p ( keys %{ $class->{function_metadata}{$sch} }) { foreach my $k (keys %{$class->{function_metadata}{$sch}{$p}}) { my $fct_name = $class->{function_metadata}{$sch}{$p}{$k}{metadata}{fct_name} || ''; next if (!$fct_name); next if ($p ne 'none' && $str !~ /\b$p\.$fct_name\b/is && $str !~ /(^|[^\.])\b$fct_name\b/is); next if ($p eq 'none' && $str !~ /\b$fct_name\b/is); my $call = 'PERFORM'; if ($class->{pg_supports_procedure} && uc($class->{function_metadata}{$sch}{$p}{$k}{metadata}{type}) eq 'PROCEDURE') { $call = 'CALL'; } if (!$class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout}) { if ($sch ne 'unknown' and $str =~ /\b$sch.$k\b/is) { # Look if we need to use $call to call the function $str =~ s/(BEGIN|LOOP|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/igs; while ($str =~ s/(EXCEPTION(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/is) {}; $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/isg; $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/isg; $str =~ s/($call $sch\.$k);/$1\(\);/igs; } elsif ($str =~ /\b($k|$fct_name)\b/is) { # Look if we need to use $call to call the function $str =~ s/(BEGIN|LOOP|CALL|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/igs; while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/is) {}; $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/isg; $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/isg; $str =~ s/($call (?:$k|$fct_name));/$1\(\);/igs; } } else { # Recover call to function with OUT parameter with double affectation $str =~ s/([^:\s]+\s*:=\s*)[^:\s]*\s+:=\s*((?:[^\s\.]+\.)?\b$fct_name\s*\()/$1$2/isg; } # Remove package name and try to replace call to function name only if (!$class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} && $k =~ s/^[^\.]+\.// && lc($p) eq lc($class->{current_package}) ) { if ($sch ne 'unknown' and $str =~ /\b$sch\.$k\b/is) { $str =~ s/(BEGIN|LOOP|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/igs; while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/is) {}; $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/isg; $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2$call $3/isg; $str =~ s/($call $sch\.$k);/$1\(\);/igs; } elsif ($str =~ /\b(?:$k|$fct_name)\b/is) { $str =~ s/(BEGIN|LOOP|CALL|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/igs; while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/is) {}; $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/isg; $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)((?:$k|$fct_name)\s*[\(;])/$1$2$call $3/isg; $str =~ s/($call (?:$k|$fct_name));/$1\(\);/igs; } } } } } } # Fix call to procedure changed above if ($class->{pg_supports_procedure}) { while ($str =~ s/\bCALL\s+(PERFORM|CALL)\s+/CALL /igs) {}; } else { while ($str =~ s/\bCALL\s+(PERFORM|CALL)\s+/PERFORM /igs) {}; } return $str; } sub translate_statement { my ($class, $stmt, $is_subpart) = @_; # Divise code through UNION keyword marking a new query level my @q = split(/\b(UNION\s+ALL|UNION)\b/i, $stmt); for (my $j = 0; $j <= $#q; $j++) { next if ($q[$j] =~ /^UNION/); # Replace call to right outer join obsolete syntax $q[$j] = replace_outer_join($class, $q[$j], 'right'); # Replace call to left outer join obsolete syntax $q[$j] = replace_outer_join($class, $q[$j], 'left'); if ($q[$j] =~ /\bROWNUM\b/i) { # Replace ROWNUM after the WHERE clause by a LIMIT clause $q[$j] = replace_rownum_with_limit($class, $q[$j]); # Replace ROWNUM by row_number() when used in the target list $q[$j] =~ s/((?!WHERE\s.*|LIMIT\s.*)[\s,]+)ROWNUM([\s,]+)/$1row_number() OVER () AS rownum$2/is; # Aliases before =, <, >, +, -, ASC or DESC will generate an error $q[$j] =~ s/row_number\(\) OVER \(\) AS rownum\s*([=><+\-]|ASC|DESC)/row_number() OVER () $1/is; # Try to replace AS rownnum with alias if there is one already defined $q[$j] =~ s/(row_number\(\) OVER \(\) AS)\s+rownum\s+((?!FROM\s+|[,+\-]\s*)[^\s]+)/$1 $2/is; $q[$j] =~ s/\s+AS(\s+AS\s+)/$1/is; # The form "UPDATE mytbl SET col1 = ROWNUM;" is not yet translated # and mus be manually rewritten as follow: # WITH cte AS (SELECT *, ROW_NUMBER() OVER() AS rn FROM mytbl) # UPDATE mytbl SET col1 = (SELECT rn FROM cte WHERE cte.pk = mytbl.pk); } } $stmt = join("\n", @q); # Rewrite some invalid ending form after rewriting $stmt =~ s/(\s+WHERE)\s+AND/$1/igs; $stmt =~ s/(\s+)(?:WHERE|AND)\s+(LIMIT\s+)/$1$2/igs; $stmt =~ s/\s+WHERE\s*$//is; $stmt =~ s/\s+WHERE\s*\)/\)/is; # Remove unnecessary offset to position 0 which is the default $stmt =~ s/\s+OFFSET 0//igs; # Replacement of connect by with CTE $stmt = replace_connect_by($class, $stmt); return $stmt; } sub remove_extra_parenthesis { my $str = shift; while ($str =~ s/\(\s*\(((?!\s*SELECT)[^\(\)]+)\)\s*\)/($1)/gs) {}; my %store_clause = (); my $i = 0; while ($str =~ s/\(\s*\(([^\(\)]+)\)\s*AND\s*\(([^\(\)]+)\)\s*\)/\%PARENTHESIS$i\%/is) { $store_clause{$i} = find_or_parenthesis($1, $2); $i++ } $str =~ s/\%PARENTHESIS(\d+)\%/$store_clause{$1}/gs; while ($str =~ s/\(\s*\(\s*\(([^\(\)]+\)[^\(\)]+\([^\(\)]+)\)\s*\)\s*\)/(($1))/gs) {}; return $str; } # When the statement include OR keep parenthesisœ sub find_or_parenthesis { my ($left, $right) = @_; if ($left =~ /\s+OR\s+/i) { $left = "($left)"; } if ($right =~ /\s+OR\s+/i) { $right = "($right)"; } return "($left AND $right)"; } sub extract_subpart { my ($class, $str) = @_; while ($$str =~ s/\(([^\(\)]*)\)/\%SUBQUERY$class->{sub_parts_idx}\%/s) { $class->{sub_parts}{$class->{sub_parts_idx}} = $1; $class->{sub_parts_idx}++; } my @done = (); foreach my $k (sort { $b <=> $a } %{$class->{sub_parts}}) { if ($class->{sub_parts}{$k} =~ /\%OUTERJOIN\d+\%/ && $class->{sub_parts}{$k} !~ /\b(SELECT|FROM|WHERE)\b/i) { $$str =~ s/\%SUBQUERY$k\%/\($class->{sub_parts}{$k}\)/s; push(@done, $k); } } foreach (@done) { delete $class->{sub_parts}{$_}; } } sub extract_subqueries { my ($class, $str) = @_; return if ($class->{sub_queries_idx} == 100); my $cur_idx = $class->{sub_queries_idx}; if ($$str =~ s/\((\s*(?:SELECT|WITH).*)/\%SUBQUERY$class->{sub_queries_idx}\%/is) { my $stop_learning = 0; my $idx = 1; my $sub_query = ''; foreach my $c (split(//, $1)) { $idx++ if (!$stop_learning && $c eq '('); $idx-- if (!$stop_learning && $c eq ')'); if ($idx == 0) { # Do not copy last parenthesis in the output string $c = '' if (!$stop_learning); # Increment storage position for the next subquery $class->{sub_queries_idx}++ if (!$stop_learning); # Inform the loop that we don't want to process any charater anymore $stop_learning = 1; # We have reach the end of the subquery all next # characters must be restored to the final string. $$str .= $c; } elsif ($idx > 0) { # Append character to the current substring storage $class->{sub_queries}{$class->{sub_queries_idx}} .= $c; } } # Each subquery could have subqueries too, so call the # function recursively on each extracted subquery if ($class->{sub_queries}{$class->{sub_queries_idx}-1} =~ /\(\s*(?:SELECT|WITH)/is) { extract_subqueries($class, \$class->{sub_queries}{$class->{sub_queries_idx}-1}); } } } sub replace_rownum_with_limit { my ($class, $str) = @_; my $offset = ''; if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { $offset = $2; ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; $class->{limit_clause} = ' LIMIT 1 OFFSET ' . $offset; } if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { $offset = $1; ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; $class->{limit_clause} = ' LIMIT 1 OFFSET ' . $offset; } if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*>=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { $offset = $2; ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; } if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*>\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { $offset = $2; $offset = "($offset)" if ($offset =~ /[^0-9]/); $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; } if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*>=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { $offset = $1; ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; } if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*>\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { $offset = $1; $offset = "($offset)" if ($offset =~ /[^0-9]/); $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; } my $tmp_val = ''; if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*<=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { $tmp_val = $2; } if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*<\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { my $clause = $2; if ($clause =~ /\%SUBQUERY\d+\%/) { $tmp_val = $clause; } elsif ($clause !~ /\D/) { $tmp_val = $clause - 1; } else { $tmp_val = "$clause - 1"; } } if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*<=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { $tmp_val = $1; } if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*<\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { my $clause = $1; if ($clause =~ /\%SUBQUERY\d+\%/) { $tmp_val = $clause; } elsif ($clause !~ /\D/) { $tmp_val = $clause - 1; } else { $tmp_val = "$clause - 1"; } } $str =~ s/\s+WHERE\s+ORDER\s+/ ORDER /is; if ($tmp_val) { if ($class->{limit_clause} =~ /LIMIT ALL OFFSET ([^\s]+)/is) { my $tmp_offset = $1; if ($tmp_offset !~ /[^0-9]/ && $tmp_val !~ /[^0-9]/) { $tmp_val -= $tmp_offset; } else { $tmp_val = "($tmp_val - $tmp_offset)"; } $class->{limit_clause} =~ s/LIMIT ALL/LIMIT $tmp_val/is; } else { $tmp_val = "($tmp_val)" if ($tmp_val =~ /[^0-9]/); $class->{limit_clause} = ' LIMIT ' . $tmp_val; } } # Rewrite some invalid ending form after rewriting $str =~ s/(\s+WHERE)\s+AND/$1/igs; $str =~ s/\s+WHERE\s*$//is; $str =~ s/\s+WHERE\s*\)/\)/is; # Remove unnecessary offset to position 0 which is the default $str =~ s/\s+OFFSET 0//igs; return $str; } # Translation of REGEX_SUBSTR( string, pattern, [pos], [nth]) converted into # (SELECT array_to_string(a, '') FROM regexp_matches(substr(string, pos), pattern, 'g') AS foo(a) LIMIT 1 OFFSET (nth - 1))"; # Optional fith parameter of match_parameter is appended to 'g' when present sub convert_regex_substr { ($class, $str) = @_; my @params = split(/\s*,\s*/, $str); my $mod = ''; if ($#params == 4) { # Restore constant string to look into date format while ($params[4] =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/igs) {}; #delete $class->{text_values}{$1}; # $params[4] =~ s/'//g; $mod = $params[4] if ($params[4] ne 'g'); } if ($#params < 2) { push(@params, 1, 1); } elsif ($#params < 3) { push(@params, 1); } if ($params[2] == 1) { $str = "(SELECT array_to_string(a, '') FROM regexp_matches($params[0], $params[1], 'g$mod') AS foo(a) LIMIT 1 OFFSET ($params[3] - 1))"; } else { $str = "(SELECT array_to_string(a, '') FROM regexp_matches(substr($params[0], $params[2]), $params[1], 'g$mod') AS foo(a) LIMIT 1 OFFSET ($params[3] - 1))"; } return $str; } sub convert_from_tz { my ($class, $date) = @_; # Restore constant string to look into date format while ($date =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/igs) {}; #delete $class->{text_values}{$1}; my $tz = '00:00'; if ($date =~ /^[^']*'([^']+)'\s*,\s*'([^']+)'/) { $date = $1; $tz = $2; $date .= ' '; if ($tz =~ /^\d+:\d+$/) { $date .= '+' . $tz; } else { $date .= $tz; } $date = "'$date'"; } elsif ($date =~ /^(.*),\s*'([^']+)'$/) { $date = $1; $tz = $2; if ($tz =~ /^\d+:\d+$/) { $tz .= '+' . $tz; } $date .= ' AT TIME ZONE ' . "'$tz'"; } # Replace constant strings while ($date =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/is) { $class->{text_values}{$class->{text_values_pos}} = $1; $class->{text_values_pos}++; } return $date; } sub convert_date_format { my ($class, $fields, @strings) = @_; # Restore constant string to look into date format while ($fields =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/igs) {}; for ($i = 0; $i <= $#strings; $i++) { $fields =~ s/\%\%string$i\%\%/'$strings[$i]'/; } # Truncate time to microsecond $fields =~ s/(\d{2}:\d{2}:\d{2}[,\.]\d{6})\d{3}/$1/s; # Replace round year with two digit year format. $fields =~ s/RR/YY/sg; # Convert fractional seconds to milli (MS) or micro (US) seconds $fields =~ s/FF[123]/MS/s; $fields =~ s/FF\d*/US/s; # Remove any timezone format if ($class->{to_char_notimezone}) { $fields =~ s/\s*TZ[DHMR]//gs; } # Replace constant strings while ($str =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/s) { $class->{text_values}{$class->{text_values_pos}} = $1; $class->{text_values_pos}++; } return $fields; } #------------------------------------------------------------------------------ # Set the correspondance between Oracle and PostgreSQL regexp modifiers # Oracle default: # 1) The default case sensitivity is determined by the NLS_SORT parameter. # Ora2pg assuming case sensitivy # 2) A period (.) does not match the newline character. # 3) The source string is treated as a single line. # PostgreSQL default: # 1) Default to case sensitivity # 2) A period match the newline character. # 3) The source string is treated as a single line. # Oracle only supports the following modifiers # 'i' specifies case-insensitive matching. Same for PG. # 'c' specifies case-sensitive matching. Same for PG. # 'x' Ignores whitespace characters in the search pattern. Same for PG. # 'n' allows the period (.) to match the newline character. PG => s. # 'm' treats the source string as multiple lines. PG => n. #------------------------------------------------------------------------------ sub regex_flags { my ($class, $modifier, $append) = @_; my $nconst = ''; my $flags = $append || ''; if ($modifier =~ /\?TEXTVALUE(\d+)\?/) { $nconst = $1; $modifier =~ s/\?TEXTVALUE$nconst\?/$class->{text_values}{$nconst}/igs; } # These flags have the same behavior if ($modifier =~ /([icx]+)/) { $flags .= $1; } # Oracle: # m : treats the source string as multiple lines. # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('Hello'||CHR(10)||'world!', '^world!$', 'm'); => 1 # PostgreSQL: # m : historical synonym for n => m : newline-sensitive matching # SELECT regexp_match('Hello'||chr(10)||'world!', '^world!$', 'm'); => match if ($modifier =~ /m/) { $flags .= 'n'; } # Oracle: # n: allows the period (.) to match the newline character. # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', 'a.d', 'n'); => 1 # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', '^d$', 'n'); => not match # PostgreSQL: # s: non-newline-sensitive matching (default) # SELECT regexp_match('a'||chr(10)||'d', 'a.d', 's'); => match # SELECT regexp_match('a'||chr(10)||'d', '^d$', 's'); => not match if ($modifier =~ /n/) { $flags .= 's'; } # By default PG is non-newline-sensitive whereas Oracle is newline-sensitive # Oracle: # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', 'a.d'); => not match # PostgreSQL: # SELECT regexp_match('a'||chr(10)||'d', 'a.d'); => match # Add 'n' to force the same behavior like Oracle $flags .= 'n' if ($flags !~ /n|s/); if ($nconst ne '') { $class->{text_values}{$nconst} = "'$flags'"; return "?TEXTVALUE$nconst?"; } return "'$flags'"; } sub replace_oracle_function { my ($class, $str, @strings) = @_; my @xmlelt = (); my $field = '\s*([^\(\),]+)\s*'; my $num_field = '\s*([\d\.]+)\s*'; # Remove the SYS schema from calls $str =~ s/\bSYS\.//igs; #-------------------------------------------- # PL/SQL to PL/PGSQL code conversion # Feel free to add your contribution here. #-------------------------------------------- if ($class->{is_mysql}) { $str = mysql_to_plpgsql($class, $str); } elsif ($class->{is_mssql}) { $str = mssql_to_plpgsql($class, $str); } # Change NVL to COALESCE $str =~ s/NVL\s*\(/coalesce(/is; $str =~ s/NVL2\s*\($field,$field,$field\)/(CASE WHEN $1 IS NOT NULL THEN $2 ELSE $3 END)/is; # Replace DEFAULT empty_blob() and empty_clob() my $empty = "''"; $empty = 'NULL' if ($class->{empty_lob_null}); $str =~ s/(empty_blob|empty_clob)\s*\(\s*\)/$empty/is; $str =~ s/(empty_blob|empty_clob)\b/$empty/is; # DBMS_LOB.GETLENGTH can be replaced by binary length. $str =~ s/DBMS_LOB.GETLENGTH/octet_length/igs; # DBMS_LOB.SUBSTR can be replaced by SUBSTR() with second and third parameter inversed $str =~ s/DBMS_LOB.SUBSTR\s*\($field,$field,$field\)/substr($1, $3, $2)/igs; # TO_CLOB(), we just remove it $str =~ s/TO_CLOB\s*\(/\(/igs; # DBMS_LOCK.SLEEP can be replaced by pg_sleep $str =~ s/DBMS_(LOCK|SESSION)\.SLEEP/pg_sleep/igs; # Replace call to SYS_GUID() function $str =~ s/\bSYS_GUID\s*\(\s*\)/$class->{uuid_function}()/igs; $str =~ s/\bSYS_GUID\b/$class->{uuid_function}()/igs; # Rewrite TO_DATE formating call # Optional nls_param parameter, if present, is removed implicitly, as it is not manageable on PostgreSQL. $str =~ s/TO_DATE\s*\(\s*('[^\']+')\s*,\s*('[^\']+')[^\)]*\)/to_date($1,$2)/igs; $str =~ s/TO_DATE\s*\(\s*('[^\']+')\s*,\s*('[^\']+')\s*,\s*('[^\']+')[^\)]*\)/to_date($1,$2)/igs; # Case where the parameters are obfuscated by function and string placeholders $str =~ s/TO_DATE\s*\(\s*(.*)\s*,\s*(\?TEXTVALUE\d+\?)[^\)]*\)/to_date($1,$2)/igs; $str =~ s/TO_DATE\s*\(\s*(.*)\s*,\s*(\?TEXTVALUE\d+\?)\s*,\s*(\?TEXTVALUE\d+\?)[^\)]*\)/to_date($1,$2)/igs; # When the date format is ISO and we have a constant we can remove the call to to_date() if ($class->{type} eq 'PARTITION' && $class->{pg_supports_partition}) { $str =~ s/to_date\(\s*('\s*\d+-\d+-\d+ \d+:\d+:\d+')\s*,\s*'[S]*YYYY-MM-DD HH24:MI:SS'[^\)]*\)/$1/igs; } # Translate to_timestamp_tz Oracle function $str =~ s/TO_TIMESTAMP(?:_TZ)?\s*\((.*)\)/'to_timestamp(' . convert_date_format($class, $1, @strings) . ')'/iegs; # Translate from_tz Oracle function $str =~ s/FROM_TZ\s*\(\s*([^\)]+)\s*\)/'(' . convert_from_tz($class,$1) . ')::timestamp with time zone'/iegs; # Replace call to trim into btrim $str =~ s/\b(TRIM\s*\()\s+/$1/igs; $str =~ s/\bTRIM\s*\(((?!BOTH)[^\(\)]*)\)/trim(both $1)/igs; # Do some transformation when Orafce is not used if (!$class->{use_orafce}) { # Replace to_nchar() without format by a simple cast to text $str =~ s/\bTO_NCHAR\s*\(\s*([^,\)]+)\)/($1)::varchar/igs; # Replace to_char() without format by a simple cast to text $str =~ s/\bTO_CHAR\s*\(\s*([^,\)]+)\)/($1)::varchar/igs; # Fix format for to_char() with format $str =~ s/\b(TO_CHAR\s*\(\s*[^,\)]+\s*),(\s*[^,\)]+\s*)\)/"$1," . convert_date_format($class, $2, @strings) . ")"/iegs; if ($class->{type} ne 'TABLE') { $str =~ s/\(([^\s]+)\)(::varchar)/$1$2/igs; } else { $str =~ s/\(([^\s]+)\)(::varchar)/($1$2)/igs; } # Change trunc(date) to date_trunc('day', field) # Oracle has trunc(number) so there will have false positive # replacement but most of the time trunc(date) is encountered. $str =~ s/\bTRUNC\s*\($field\)/date_trunc('day', $1)/is; if ($str =~ s/\bTRUNC\s*\($field,$field\)/date_trunc($2, $1)/is || # Case where the parameters are obfuscated by function and string placeholders $str =~ s/\bTRUNC\((\%\%REPLACEFCT\d+\%\%)\s*,\s*(\?TEXTVALUE\d+\?)\)/date_trunc($2, $1)/is ) { if ($str =~ /date_trunc\(\?TEXTVALUE(\d+)\?/) { my $k = $1; $class->{text_values}{$k} =~ s/'(SYYYY|SYEAR|YEAR|[Y]+)'/'year'/is; $class->{text_values}{$k} =~ s/'Q'/'quarter'/is; $class->{text_values}{$k} =~ s/'(MONTH|MON|MM|RM)'/'month'/is; $class->{text_values}{$k} =~ s/'(IW|DAY|DY|D)'/'week'/is; $class->{text_values}{$k} =~ s/'(DDD|DD|J)'/'day'/is; $class->{text_values}{$k} =~ s/'(HH|HH12|HH24)'/'hour'/is; $class->{text_values}{$k} =~ s/'MI'/'minute'/is; } } # Convert the call to the Oracle function add_months() into Pg syntax $str =~ s/\bADD_MONTHS\s*\(([^,]+),\s*(\d+)\s*\)/$1 + '$2 month'::interval/si; $str =~ s/\bADD_MONTHS\s*\(([^,]+),\s*([^,\(\)]+)\s*\)/$1 + $2*'1 month'::interval/si; # Convert the call to the Oracle function add_years() into Pg syntax $str =~ s/\bADD_YEARS\s*\(([^,]+),\s*(\d+)\s*\)/$1 + '$2 year'::interval/si; $str =~ s/\bADD_YEARS\s*\(([^,]+),\s*([^,\(\)]+)\s*\)/$1 + $2*' year'::interval/si; # Translate numtodsinterval Oracle function $str =~ s/\b(?:NUMTODSINTERVAL|NUMTOYMINTERVAL)\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/($1 * ('1'||$2)::interval)/is; # REGEX_LIKE( string, pattern, flags ) $str =~ s/\bREGEXP_LIKE\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\)]+)\s*\)/"regexp_match($1, $2," . regex_flags($class, $3) . ") IS NOT NULL"/iges; # REGEX_LIKE( string, pattern ) $str =~ s/\bREGEXP_LIKE\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/"regexp_match($1, $2," . regex_flags($class, '') . ") IS NOT NULL"/iges; # REGEX_COUNT( string, pattern, position, flags ) $str =~ s/\bREGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*(\d+)\s*,\s*([^\)]+)\s*\)/"(SELECT count(*) FROM regexp_matches(substr($1, $3), $2, " . regex_flags($class, $4, 'g') . "))"/iges; # REGEX_COUNT( string, pattern, position ) $str =~ s/\bREGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*(\d+)\s*\)/(SELECT count(*) FROM regexp_matches(substr($1, $3), $2, 'g'))/igs; # REGEX_COUNT( string, pattern ) $str =~ s/\bREGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/(SELECT count(*) FROM regexp_matches($1, $2, 'g'))/igs; # REGEX_SUBSTR( string, pattern, pos, num ) translation $str =~ s/\bREGEXP_SUBSTR\s*\(\s*([^\)]+)\s*\)/convert_regex_substr($class, $1)/iges; # Always append 'g' modifier to regexp_replace, this is the default with Oracle $str =~ s/\b(REGEXP_REPLACE\s*\(\s*[^\)]+)\s*\)/$1, 'g')/igs; # LAST_DAY( date ) translation $str =~ s/\bLAST_DAY\s*\(\s*([^\(\)]+)\s*\)/((date_trunc('month',($1)::timestamp + interval '1 month'))::date - 1)/igs; # replace the BITAND function by the & operator $str =~ s/BITAND\(\s*([^,\(]+)\s*,\s*([^\)]+)\s*\)/($1 & $2)/igs; } else { s/([^\.])\b(GREATEST\()/$1oracle.$2/igs; s/([^\.])\b(LEAST\()/$1oracle.$2/igs; } # Replace INSTR by POSITION $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*([^\),]+)\s*\)/position($2 in $1)/is; $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*([^,]+)\s*,\s*1\s*\)/position($2 in $1)/is; # The to_number() function reclaim a second argument under postgres which is the format. # Replace to_number with a cast when no specific format is given if (lc($class->{to_number_conversion}) ne 'none') { if ($class->{to_number_conversion} =~ /(numeric|bigint|integer|int)/i) { my $cast = lc($1); if ($class->{type} ne 'TABLE' && $class->{type} ne 'COPY') { $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)\s?/($1)\:\:$cast /is; } else { $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)\s?/(nullif($1, '')\:\:$cast) /is; } } else { $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)/to_number\($1,'$class->{to_number_conversion}'\)/is; } } # Replace the UTC convertion with the PG syntaxe $str =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/($1 AT TIME ZONE 'UTC')/is; # Remove call to XMLCDATA, there's no such function with PostgreSQL $str =~ s/XMLCDATA\s*\(([^\)]+)\)/''/is; # Remove call to getClobVal() or getStringVal, no need of that $str =~ s/\.(GETCLOBVAL|GETSTRINGVAL|GETNUMBERVAL|GETBLOBVAL)\s*\(\s*\)//is; # Add the name keyword to XMLELEMENT $str =~ s/XMLELEMENT\s*\(\s*(?:NAME\s+)?([^\)]+)/XMLELEMENT(name $1/is; # Replace XMLTYPE function $str =~ s/XMLTYPE\s*\(\s*([^,]+)\s*,[^\)]+\s*\)/xmlparse(DOCUMENT, convert_from($1, 'utf-8'))/igs; $str =~ s/XMLTYPE\.CREATEXML\s*\(\s*[^\)]+\s*\)/xmlparse(DOCUMENT, convert_from($1, 'utf-8'))/igs; # Cast round() call as numeric $str =~ s/round\s*\(([^,]+),([\s\d]+)\)/round\(($1)::numeric,$2\)/is; if ($str =~ /SDO_/is) { # Replace SDO_GEOM to the postgis equivalent $str = &replace_sdo_function($str); # Replace Spatial Operator to the postgis equivalent $str = &replace_sdo_operator($str); } # Rewrite replace(a,b) with three argument $str =~ s/REPLACE\s*\($field,$field\)/replace($1, $2, '')/is; # Replace Oracle substr(string, start_position, length) with # PostgreSQL substring(string from start_position for length) $str =~ s/\bsubstrb\s*\(/substr\(/igs; if (!$class->{pg_supports_substr}) { $str =~ s/\bsubstr\s*\($field,$field,$field\)/substring($1 from $2 for $3)/is; $str =~ s/\bsubstr\s*\($field,$field\)/substring($1 from $2)/is; } # Replace call to function with out parameters $str = replace_out_param_call($class, $str); # Replace some sys_context call to the postgresql equivalent if ($str =~ /SYS_CONTEXT/is) { $str = replace_sys_context($str); } return $str; } sub replace_out_param_call_internal { my ($class, $fct_name, $str, $sch, $p, $k) = @_; my %replace_out_parm = (); my $idx = 0; while ($str =~ s/((?:[^\s\.]+\.)?\b$fct_name)\s*\(([^\(\)]+)\)/\%FCTINOUTPARAM$idx\%/is) { my $fname = $1; my $fparam = $2; if ($fname =~ /\./ && lc($fname) ne lc($k)) { $replace_out_parm{$idx} = "$fname($fparam)"; next; } $replace_out_parm{$idx} = "$fname("; # Extract position of out parameters my @params = split(/\s*,\s*/, $class->{function_metadata}{$sch}{$p}{$k}{metadata}{args}); my @cparams = split(/\s*,\s*/s, $fparam); my $call_params = ''; my @out_pos = (); my @out_fields = (); for (my $i = 0; $i <= $#params; $i++) { if (!$class->{is_mysql} && $params[$i] =~ /\s*([^\s]+)\s+(OUT|INOUT)\s/is) { push(@out_fields, $1); push(@out_pos, $i); $call_params .= $cparams[$i] if ($params[$i] =~ /\bINOUT\b/is); } elsif ($class->{is_mysql} && $params[$i] =~ /\s*(OUT|INOUT)\s+([^\s]+)\s/is) { push(@out_fields, $2); push(@out_pos, $i); $call_params .= $cparams[$i] if ($params[$i] =~ /\bINOUT\b/is); } else { $call_params .= $cparams[$i]; } $call_params .= ', ' if ($i < $#params); } map { s/^\(//; } @out_fields; $call_params =~ s/(\s*,\s*)+$//s; while ($call_params =~ s/\s*,\s*,\s*/, /s) {}; $call_params =~ s/^(\s*,\s*)+//s; $replace_out_parm{$idx} .= "$call_params)"; my @out_param = (); foreach my $i (@out_pos) { push(@out_param, $cparams[$i]); } if ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} == 1) { map { s/[^\s=]+\s*=>\s*//; } @out_param; if ($#out_param == 0) { $replace_out_parm{$idx} = "$out_param[0] := $replace_out_parm{$idx}"; } else { $replace_out_parm{$idx} = "SELECT * FROM $replace_out_parm{$idx} INTO " . join(', ', @out_param); } } elsif ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} > 1) { $class->{replace_out_params} = "_ora2pg_r RECORD;" if (!$class->{replace_out_params}); $replace_out_parm{$idx} = "SELECT * FROM $replace_out_parm{$idx} INTO _ora2pg_r;\n"; my $out_field_pos = 0; foreach my $parm (@out_param) { # remove use of named parameters $parm =~ s/.*=>\s*//; $replace_out_parm{$idx} .= " $parm := _ora2pg_r.$out_fields[$out_field_pos++];"; } $replace_out_parm{$idx} =~ s/;$//s; } $idx++; } $str =~ s/\%FCTINOUTPARAM(\d+)\%/$replace_out_parm{$1}/gs; return $str; } ############## # Replace call to function with out parameters ############## sub replace_out_param_call { my ($class, $str) = @_; if (uc($class->{type}) =~ /^(PACKAGE|FUNCTION|PROCEDURE|TRIGGER)$/) { foreach my $sch (sort keys %{$class->{function_metadata}}) { foreach my $p (sort keys %{$class->{function_metadata}{$sch}}) { foreach my $k (sort keys %{$class->{function_metadata}{$sch}{$p}}) { if ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout}) { my $fct_name = $class->{function_metadata}{$sch}{$p}{$k}{metadata}{fct_name} || ''; next if (!$fct_name); next if ($p eq 'none' && $str !~ /\b$fct_name\b/is); next if ($p ne 'none' && $str !~ /\b$p\.$fct_name\b/is && $str !~ /(^|[^\.])\b$fct_name\b/is); # Prevent replacement with same function name from an other package next if ($class->{current_package} && lc($p) ne lc($class->{current_package}) && $str =~ /(^|[^\.])\b$fct_name\b/is); # Since PG14 procedures support OUT param should not be # changed, just add CALL at start of the function call if ($class->{pg_supports_outparam} && $class->{function_metadata}{$sch}{$p}{$k}{metadata}{type} eq 'PROCEDURE') { $str =~ s/(^|\s+)($fct_name)\b/$1 CALL $2/igs; $str =~ s/\b($p\.$fct_name)\b/CALL $1/igs; next; } $str = &replace_out_param_call_internal($class, $fct_name, $str, $sch, $p, $k); } } } } # Replace regular procedur call (not proc from package) foreach my $sch (sort keys %{$class->{function_metadata}}) { my $p = 'none'; foreach my $k (sort keys %{$class->{function_metadata}{$sch}{$p}}) { if ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout}) { my $fct_name = $class->{function_metadata}{$sch}{$p}{$k}{metadata}{fct_name} || ''; next if (!$fct_name); # Prevent replacement with same function name from an other package next if ($str !~ /(^|[\.])\b$fct_name\b/is); # Since PG14 procedures support OUT param should not be # changed, just add CALL at start of the function call if ($class->{pg_supports_outparam} && $class->{function_metadata}{$sch}{$p}{$k}{metadata}{type} eq 'PROCEDURE') { $str =~ s/(^|\s+)($fct_name)\b/$1 CALL $2/igs; $str =~ s/\b($p\.$fct_name)\b/CALL $1/igs; next; } $str = &replace_out_param_call_internal($class, $fct_name, $str, $sch, $p, $k); } } } } return $str; } # Replace decode("user_status",'active',"username",null) # PostgreSQL (CASE WHEN "user_status"='ACTIVE' THEN "username" ELSE NULL END) sub replace_decode { my $str = shift; while ($str =~ s/\bDECODE\s*\((.*)$/\%DECODE\%/is) { my @decode_params = (''); my $stop_learning = 0; my $idx = 1; foreach my $c (split(//, $1)) { $idx++ if (!$stop_learning && $c eq '('); $idx-- if (!$stop_learning && $c eq ')'); if ($idx == 0) { # Do not copy last parenthesis in the output string $c = '' if (!$stop_learning); # Inform the loop that we don't want to process any charater anymore $stop_learning = 1; # We have reach the end of the decode() parameter # next character must be restored to the final string. $str .= $c; } elsif ($idx > 0) { # We are parsing the decode() parameter part, append # the caracter to the right part of the param array. if ($c eq ',' && ($idx - 1) == 0) { # we are switching to a new parameter push(@decode_params, ''); } elsif ($c ne "\n") { $decode_params[-1] .= $c; } } } my $case_str = 'CASE '; for (my $i = 1; $i <= $#decode_params; $i+=2) { $decode_params[$i] =~ s/^\s+//gs; $decode_params[$i] =~ s/\s+$//gs; if ($i < $#decode_params) { $case_str .= "WHEN $decode_params[0]=$decode_params[$i] THEN $decode_params[$i+1] "; } else { $case_str .= " ELSE $decode_params[$i] "; } } $case_str .= 'END '; $str =~ s/\%DECODE\%/$case_str/s; } return $str; } # Function to replace call to SYS_CONTECT('USERENV', ...) # List of Oracle environment variables: http://docs.oracle.com/cd/B28359_01/server.111/b28286/functions172.htm # Possibly corresponding PostgreSQL variables: http://www.postgresql.org/docs/current/static/functions-info.html sub replace_sys_context { my $str = shift; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(OS_USER|SESSION_USER|AUTHENTICATED_IDENTITY)'\s*\)/session_user/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'BG_JOB_ID'\s*\)/pg_backend_pid()/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(CLIENT_IDENTIFIER|PROXY_USER)'\s*\)/session_user/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'CURRENT_SCHEMA'\s*\)/current_schema/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'CURRENT_USER'\s*\)/current_user/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(DB_NAME|DB_UNIQUE_NAME)'\s*\)/current_database/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(HOST|IP_ADDRESS)'\s*\)/inet_client_addr()/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'SERVER_HOST'\s*\)/inet_server_addr()/is; $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'MODULE'\s*\)/current_setting('application_name')/is; return $str; } sub replace_sdo_function { my $str = shift; $str =~ s/SDO_GEOM\.RELATE/ST_Relate/igs; $str =~ s/SDO_GEOM\.VALIDATE_GEOMETRY_WITH_CONTEXT/ST_IsValidReason/igs; $str =~ s/SDO_GEOM\.WITHIN_DISTANCE/ST_DWithin/igs; $str =~ s/SDO_GEOM\.//igs; $str =~ s/SDO_DISTANCE/ST_Distance/igs; $str =~ s/SDO_BUFFER/ST_Buffer/igs; $str =~ s/SDO_CENTROID/ST_Centroid/igs; $str =~ s/SDO_UTIL\.GETVERTICES/ST_DumpPoints/igs; $str =~ s/SDO_TRANSLATE/ST_Translate/igs; $str =~ s/SDO_SIMPLIFY/ST_Simplify/igs; $str =~ s/SDO_AREA/ST_Area/igs; $str =~ s/SDO_CONVEXHULL/ST_ConvexHull/igs; $str =~ s/SDO_DIFFERENCE/ST_Difference/igs; $str =~ s/SDO_INTERSECTION/ST_Intersection/igs; $str =~ s/SDO_LENGTH/ST_Length/igs; $str =~ s/SDO_POINTONSURFACE/ST_PointOnSurface/igs; $str =~ s/SDO_UNION/ST_Union/igs; $str =~ s/SDO_XOR/ST_SymDifference/igs; # SDO_CS.TRANSFORM(geom, srid) $str =~ s/\bSDO_CS\.TRANSFORM\(/ST_Transform\(/igs; # Note that with ST_DumpPoints and : # TABLE(SDO_UTIL.GETVERTICES(C.GEOLOC)) T # T.X, T.Y, T.ID must be replaced manually as ST_X(T.geom) X, ST_Y(T.geom) Y, (T).path[1] ID my $field = '\s*[^\(\),]+\s*'; my $num_field = '\s*[\d\.]+\s*'; # SDO_GEOM.RELATE(geom1 IN SDO_GEOMETRY,mask IN VARCHAR2,geom2 IN SDO_GEOMETRY,tol IN NUMBER) $str =~ s/(ST_Relate\s*\($field),$field,($field),($field)\)/$1,$2\)/is; # SDO_GEOM.RELATE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,mask IN VARCHAR2,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) $str =~ s/(ST_Relate\s*\($field),$field,$field,($field),$field\)/$1,$2\)/is; # SDO_GEOM.SDO_AREA(geom IN SDO_GEOMETRY, tol IN NUMBER [, unit IN VARCHAR2]) # SDO_GEOM.SDO_AREA(geom IN SDO_GEOMETRY,dim IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) $str =~ s/(ST_Area\s*\($field),[^\)]+\)/$1\)/is; # SDO_GEOM.SDO_BUFFER(geom IN SDO_GEOMETRY,dist IN NUMBER, tol IN NUMBER [, params IN VARCHAR2]) $str =~ s/(ST_Buffer\s*\($field,$num_field),[^\)]+\)/$1\)/is; # SDO_GEOM.SDO_BUFFER(geom IN SDO_GEOMETRY,dim IN SDO_DIM_ARRAY,dist IN NUMBER [, params IN VARCHAR2]) $str =~ s/(ST_Buffer\s*\($field),$field,($num_field)[^\)]*\)/$1,$2\)/is; # SDO_GEOM.SDO_CENTROID(geom1 IN SDO_GEOMETRY,tol IN NUMBER) # SDO_GEOM.SDO_CENTROID(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY) $str =~ s/(ST_Centroid\s*\($field),$field\)/$1\)/is; # SDO_GEOM.SDO_CONVEXHULL(geom1 IN SDO_GEOMETRY,tol IN NUMBER) # SDO_GEOM.SDO_CONVEXHULL(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY) $str =~ s/(ST_ConvexHull\s*\($field),$field\)/$1\)/is; # SDO_GEOM.SDO_DIFFERENCE(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER) $str =~ s/(ST_Difference\s*\($field,$field),$field\)/$1\)/is; # SDO_GEOM.SDO_DIFFERENCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) $str =~ s/(ST_Difference\s*\($field),$field,($field),$field\)/$1,$2\)/is; # SDO_GEOM.SDO_DISTANCE(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER [, unit IN VARCHAR2]) $str =~ s/(ST_Distance\s*\($field,$field),($num_field)[^\)]*\)/$1\)/is; # SDO_GEOM.SDO_DISTANCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) $str =~ s/(ST_Distance\s*\($field),$field,($field),($field)[^\)]*\)/$1,$2\)/is; # SDO_GEOM.SDO_INTERSECTION(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER) $str =~ s/(ST_Intersection\s*\($field,$field),$field\)/$1\)/is; # SDO_GEOM.SDO_INTERSECTION(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) $str =~ s/(ST_Intersection\s*\($field),$field,($field),$field\)/$1,$2\)/is; # SDO_GEOM.SDO_LENGTH(geom IN SDO_GEOMETRY, dim IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) # SDO_GEOM.SDO_LENGTH(geom IN SDO_GEOMETRY, tol IN NUMBER [, unit IN VARCHAR2]) $str =~ s/(ST_Length\s*\($field),($field)[^\)]*\)/$1\)/is; # SDO_GEOM.SDO_POINTONSURFACE(geom1 IN SDO_GEOMETRY, tol IN NUMBER) # SDO_GEOM.SDO_POINTONSURFACE(geom1 IN SDO_GEOMETRY, dim1 IN SDO_DIM_ARRAY) $str =~ s/(ST_PointOnSurface\s*\($field),$field\)/$1\)/is; # SDO_GEOM.SDO_UNION(geom1 IN SDO_GEOMETRY, geom2 IN SDO_GEOMETRY, tol IN NUMBER) $str =~ s/(ST_Union\s*\($field,$field),$field\)/$1\)/is; # SDO_GEOM.SDO_UNION(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) $str =~ s/(ST_Union\s*\($field),$field,($field),$field\)/$1,$2\)/is; # SDO_GEOM.SDO_XOR(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY, tol IN NUMBER) $str =~ s/(ST_SymDifference\s*\($field,$field),$field\)/$1\)/is; # SDO_GEOM.SDO_XOR(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) $str =~ s/(ST_SymDifference\s*\($field),$field,($field),$field\)/$1,$2\)/is; # SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(geom1 IN SDO_GEOMETRY, tol IN NUMBER) # SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(geom1 IN SDO_GEOMETRY, dim1 IN SDO_DIM_ARRAY) $str =~ s/(ST_IsValidReason\s*\($field),$field\)/$1\)/is; # SDO_GEOM.WITHIN_DISTANCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,dist IN NUMBER,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY [, units IN VARCHAR2]) $str =~ s/(ST_DWithin\s*\($field),$field,($field),($field),($field)[^\)]*\)/$1,$3,$2\)/is; # SDO_GEOM.WITHIN_DISTANCE(geom1 IN SDO_GEOMETRY,dist IN NUMBER,geom2 IN SDO_GEOMETRY, tol IN NUMBER [, units IN VARCHAR2]) $str =~ s/(ST_DWithin\s*\($field)(,$field)(,$field),($field)[^\)]*\)/$1$3$2\)/is; return $str; } sub replace_sdo_operator { my $str = shift; # SDO_CONTAINS(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_CONTAINS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Contains($1)/is; $str =~ s/SDO_CONTAINS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Contains($1)/is; $str =~ s/SDO_CONTAINS\s*\(([^\)]+)\)/ST_Contains($1)/is; # SDO_RELATE(geometry1, geometry2, param) = 'TRUE' $str =~ s/SDO_RELATE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Relate($1)/is; $str =~ s/SDO_RELATE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Relate($1)/is; $str =~ s/SDO_RELATE\s*\(([^\)]+)\)/ST_Relate($1)/is; # SDO_WITHIN_DISTANCE(geometry1, aGeom, params) = 'TRUE' $str =~ s/SDO_WITHIN_DISTANCE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_DWithin($1)/is; $str =~ s/SDO_WITHIN_DISTANCE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_DWithin($1)/is; $str =~ s/SDO_WITHIN_DISTANCE\s*\(([^\)]+)\)/ST_DWithin($1)/is; # SDO_TOUCH(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_TOUCH\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Touches($1)/is; $str =~ s/SDO_TOUCH\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Touches($1)/is; $str =~ s/SDO_TOUCH\s*\(([^\)]+)\)/ST_Touches($1)/is; # SDO_OVERLAPS(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_OVERLAPS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Overlaps($1)/is; $str =~ s/SDO_OVERLAPS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Overlaps($1)/is; $str =~ s/SDO_OVERLAPS\s*\(([^\)]+)\)/ST_Overlaps($1)/is; # SDO_INSIDE(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_INSIDE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Within($1)/is; $str =~ s/SDO_INSIDE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Within($1)/is; $str =~ s/SDO_INSIDE\s*\(([^\)]+)\)/ST_Within($1)/is; # SDO_EQUAL(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_EQUAL\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Equals($1)/is; $str =~ s/SDO_EQUAL\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Equals($1)/is; $str =~ s/SDO_EQUAL\s*\(([^\)]+)\)/ST_Equals($1)/is; # SDO_COVERS(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_COVERS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Covers($1)/is; $str =~ s/SDO_COVERS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Covers($1)/is; $str =~ s/SDO_COVERS\s*\(([^\)]+)\)/ST_Covers($1)/is; # SDO_COVEREDBY(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_COVEREDBY\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_CoveredBy($1)/is; $str =~ s/SDO_COVEREDBY\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_CoveredBy($1)/is; $str =~ s/SDO_COVEREDBY\s*\(([^\)]+)\)/ST_CoveredBy($1)/is; # SDO_ANYINTERACT(geometry1, geometry2) = 'TRUE' $str =~ s/SDO_ANYINTERACT\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Intersects($1)/is; $str =~ s/SDO_ANYINTERACT\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Intersects($1)/is; $str =~ s/SDO_ANYINTERACT\s*\(([^\)]+)\)/ST_Intersects($1)/is; return $str; } # Function used to rewrite dbms_output.put, dbms_output.put_line and # dbms_output.new_line by a plpgsql code sub raise_output { my ($class, $str) = @_; my @strings = split(/\s*\|\|\s*/s, $str); my @params = (); my @pattern = (); foreach my $el (@strings) { $el =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/igs; $el =~ s/ORA2PG_ESCAPE2_QUOTE/''/gs; $el =~ s/ORA2PG_ESCAPE1_QUOTE'/\\'/gs; if ($el =~ /^\s*'(.*)'\s*$/s) { push(@pattern, $1); } else { push(@pattern, '%'); push(@params, $el); } } #my $ret = "RAISE NOTICE '$pattern'"; my $ret = "'" . join('', @pattern) . "'"; $ret =~ s/\%\%/\% \%/gs; if ($#params >= 0) { $ret .= ', ' . join(', ', @params); } return 'RAISE NOTICE ' . $ret; } sub replace_sql_type { my ($self, $str) = @_; # Remove the SYS schema from type name $str =~ s/\bSYS\.//igs; $str =~ s/with local time zone/with time zone/igs; $str =~ s/([A-Z])\%ORA2PG_COMMENT/$1 \%ORA2PG_COMMENT/igs; # Replace MySQL type UNSIGNED in cast $str =~ s/\bTINYINT\s+UNSIGNED\b/smallint/igs; $str =~ s/\bSMALLINT\s+UNSIGNED\b/integer/igs; $str =~ s/\bMEDIUMINT\s+UNSIGNED\b/integer/igs; $str =~ s/\bBIGINT\s+UNSIGNED\b/numeric/igs; $str =~ s/\bINT\s+UNSIGNED\b/bigint/igs; # Replace SYS_REFCURSOR as Pg REFCURSOR $str =~ s/\bSYS_REFCURSOR\b/REFCURSOR/isg; # Remove precision for RAW|BLOB as type modifier is not allowed for type "bytea" $str =~ s/\b(RAW|BLOB)\s*\(\s*\d+\s*\)/$1/igs; # Replace type with precision my @ora_type = keys %{$self->{data_type}}; map { s/\(/\\\(/; s/\)/\\\)/; } @ora_type; my $oratype_regex = join('|', @ora_type); while ($str =~ /(.*)\b($oratype_regex)\s*\(([^\)]+)\)/i) { my $backstr = $1; my $type = uc($2); my $args = $3; # Remove extra CHAR or BYTE information from column type $args =~ s/\s*(CHAR|BYTE)\s*$//i; if ($backstr =~ /_$/) { $str =~ s/\b($oratype_regex)\s*\(([^\)]+)\)/$1\%\|$2\%\|\%/is; next; } my ($precision, $scale) = split(/\s*,\s*/, $args); $precision = 38 if ($precision eq '*'); # case of NUMBER(*,10) or NUMBER(*) $len = $precision if ($len eq '*'); $scale ||= 0; my $len = $precision || 0; $len =~ s/\D//; if ( $type =~ /CHAR|STRING/i ) { # Type CHAR have default length set to 1 # Type VARCHAR(2) must have a specified length $len = 1 if (!$len && (($type eq "CHAR") || ($type eq "NCHAR"))); $str =~ s/\b$type\b\s*\([^\)]+\)/$self->{data_type}{$type}\%\|$len\%\|\%/is; } elsif ($type =~ /TIMESTAMP/i) { $len = 6 if ($len > 6); $str =~ s/\b$type\b\s*\([^\)]+\)/timestamp\%\|$len%\|\%/is; } elsif ($type =~ /INTERVAL/i) { # Interval precision for year/month/day is not supported by PostgreSQL $str =~ s/(INTERVAL\s+YEAR)\s*\(\d+\)/$1/is; $str =~ s/(INTERVAL\s+YEAR\s+TO\s+MONTH)\s*\(\d+\)/$1/is; $str =~ s/(INTERVAL\s+DAY)\s*\(\d+\)/$1/is; # maximum precision allowed for seconds is 6 if ($str =~ /INTERVAL\s+DAY\s+TO\s+SECOND\s*\((\d+)\)/) { if ($1 > 6) { $str =~ s/(INTERVAL\s+DAY\s+TO\s+SECOND)\s*\(\d+\)/$1(6)/i; } } } elsif ($type eq "NUMBER") { # This is an integer if (!$scale) { if ($precision) { if ($self->{pg_integer_type}) { if ($precision < 5) { $str =~ s/\b$type\b\s*\([^\)]+\)/smallint/is; } elsif ($precision <= 9) { $str =~ s/\b$type\b\s*\([^\)]+\)/integer/is; } elsif ($precision <= 19) { $str =~ s/\b$type\b\s*\([^\)]+\)/bigint/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/numeric($precision)/is; } } else { $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$precision\%\|\%/i; } } elsif ($self->{pg_integer_type}) { my $tmp = $self->{default_numeric} || 'bigint'; $str =~ s/\b$type\b\s*\([^\)]+\)/$tmp/is; } } else { if ($self->{pg_numeric_type}) { if ($precision eq '') { $str =~ s/\b$type\b\s*\([^\)]+\)/decimal(38, $scale)/is; } elsif ($precision <= 6) { $str =~ s/\b$type\b\s*\([^\)]+\)/real/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/double precision/is; } } else { if ($precision eq '') { $str =~ s/\b$type\b\s*\([^\)]+\)/decimal(38, $scale)/is; } else { $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$precision,$scale\%\|\%/is; } } } } elsif ($type eq "NUMERIC") { $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$args\%\|\%/is; } elsif ( ($type eq "DEC") || ($type eq "DECIMAL") ) { $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$args\%\|\%/is; } else { # Prevent from infinit loop $str =~ s/\(/\%\|/s; $str =~ s/\)/\%\|\%/s; } } $str =~ s/\%\|\%/\)/gs; $str =~ s/\%\|/\(/gs; # Replace datatype without precision my $number = $self->{data_type}{'NUMBER'}; $number = $self->{default_numeric} if ($self->{pg_integer_type}); $str =~ s/\bNUMBER\b/$number/igs; # Set varchar without length to text $str =~ s/\bVARCHAR2\b/VARCHAR/igs; $str =~ s/\bSTRING\b/VARCHAR/igs; if ($self->{varchar_to_text}) { $str =~ s/\bVARCHAR\b(\s*(?!\())/text$1/igs; } else { $str =~ s/\bVARCHAR\b(\s*(?!\())/varchar$1/igs; } foreach my $t ('DATE','LONG RAW','LONG','NCLOB','CLOB','BLOB','BFILE','RAW','ROWID','UROWID','FLOAT','DOUBLE PRECISION','INTEGER','INT','REAL','SMALLINT','BINARY_FLOAT','BINARY_DOUBLE','BINARY_INTEGER','BOOLEAN','XMLTYPE','SDO_GEOMETRY','PLS_INTEGER','NUMBER') { if ($t eq 'DATE') { $str =~ s/\b$t\s*\(\d\)/$self->{data_type}{$t}/igs; } elsif ($t eq 'NUMBER') { if ($self->{pg_integer_type}) { my $tmp = $self->{default_numeric} || 'bigint'; $str =~ s/\b$t\b/$tmp/igs; next; } } $str =~ s/\b$t\b/$self->{data_type}{$t}/igs; } # Translate cursor declaration $str = replace_cursor_def($str); $str =~ s/;[ ]+/;/gs; return $str; } sub replace_cursor_def { my $str = shift; # Remove IN information from cursor declaration while ($str =~ s/(\bCURSOR\b[^\(]+)\(([^\)]+\bIN\b[^\)]+)\)\s*(IS|AS)/$1\(\%\%CURSORREPLACE\%\%\) IS/is) { my $args = $2; $args =~ s/\bIN\b//igs; $str =~ s/\%\%CURSORREPLACE\%\%/$args/is; } # Replace %ROWTYPE ref cursor $str =~ s/\bTYPE\s+([^\s]+)\s+(IS\s+REF\s+CURSOR|REFCURSOR)\s+RETURN\s+[^\s\%]+\%ROWTYPE;/$1 REFCURSOR;/isg; # Replace local type ref cursor my %locatype = (); my $i = 0; while ($str =~ s/\bTYPE\s+([^\s]+)\s+(IS\s+REF\s+CURSOR|REFCURSOR)\s*;/\%LOCALTYPE$i\%/is) { $localtype{$i} = "TYPE $1 IS REF CURSOR;"; my $local_type = $1; if ($str =~ s/\b([^\s]+)\s+$local_type\s*;/$1 REFCURSOR;/igs) { $str =~ s/\%LOCALTYPE$i\%//igs; } $i++; } $str =~ s/\%LOCALTYPE(\d+)\%/$localtype{$1}/gs; # Retrieve cursor names #my @cursor_names = $str =~ /\bCURSOR\b\s*([A-Z0-9_\$]+)/isg; # Reorder cursor declaration $str =~ s/\bCURSOR\b\s*([A-Z0-9_\$]+)/$1 CURSOR/isg; # Replace call to cursor type if any #foreach my $c (@cursor_names) { # $str =~ s/\b$c\%ROWTYPE/RECORD/isg; #} # Replace REF CURSOR as Pg REFCURSOR $str =~ s/\bIS(\s*)REF\s+CURSOR/REFCURSOR/isg; $str =~ s/\bREF\s+CURSOR/REFCURSOR/isg; # Replace SYS_REFCURSOR as Pg REFCURSOR $str =~ s/\bSYS_REFCURSOR\b/REFCURSOR/isg; # Replace CURSOR IS SELECT by CURSOR FOR SELECT $str =~ s/\bCURSOR(\s+)IS([\s\(]*)(\%ORA2PG_COMMENT\d+\%)?([\s\(]*)SELECT/CURSOR$1FOR$2$3$4SELECT/isg; # Replace CURSOR (param) IS SELECT by CURSOR FOR SELECT $str =~ s/\bCURSOR(\s*\([^\)]+\)\s*)IS([\s\(]*)(\%ORA2PG_COMMENT\d+\%)?([\s\(]*)SELECT/CURSOR$1FOR$2$3$4SELECT/isg; # Replace REF CURSOR as Pg REFCURSOR $str =~ s/\bIS(\s*)REF\s+CURSOR/REFCURSOR/isg; $str =~ s/\bREF\s+CURSOR/REFCURSOR/isg; # Replace SYS_REFCURSOR as Pg REFCURSOR $str =~ s/\bSYS_REFCURSOR\b/REFCURSOR/isg; # Replace OPEN cursor FOR with dynamic query $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)((?:[^;]+?)\bUSING\b)/$1 EXECUTE$2/isg; $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)\s+([^\s]+\s*;)/$1 EXECUTE $2/isg; $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)\s+(?!(\s+|SELECT|EXECUTE|WITH|\%ORA2PG_COMMENT))/$1 EXECUTE /isg; # Remove empty parenthesis after an open cursor $str =~ s/(OPEN\s+[^\(\s;]+)\s*\(\s*\)/$1/isg; # Invert FOR CURSOR call $str =~ s/\bFOR\s+CURSOR\s*\(([^;]+)?\);/CURSOR FOR $1;/igs; $str =~ s/\bFOR\s+CURSOR(\s+)/CURSOR FOR$1/igs; return $str; } sub estimate_cost { my ($class, $str, $type) = @_; return mysql_estimate_cost($class, $str, $type) if ($class->{is_mysql}); return mssql_estimate_cost($class, $str, $type) if ($class->{is_mssql}); my %cost_details = (); # Remove some unused pragma from the cost assessment $str =~ s/PRAGMA RESTRICT_REFERENCES[^;]+;//igs; $str =~ s/PRAGMA SERIALLY_REUSABLE[^;]*;//igs; $str =~ s/PRAGMA INLINE[^;]+;//igs; # Default cost is testing that mean it at least must be tested my $cost = $FCT_TEST_SCORE; # When evaluating queries size must not be included here if ($type eq 'QUERY' || $type eq 'VIEW') { $cost = 0; } $cost_details{'TEST'} = $cost; # Set cost following code length my $cost_size = int(length($str)/$SIZE_SCORE) || 1; # When evaluating queries size must not be included here if ($type eq 'QUERY' || $type eq 'VIEW') { $cost_size = 0; } $cost += $cost_size; $cost_details{'SIZE'} = $cost_size; # Try to figure out the manual work my $n = () = $str =~ m/\bIS\s+TABLE\s+OF\b/igs; $cost_details{'IS TABLE OF'} += $n; $n = () = $str =~ m/\(\+\)/igs; $cost_details{'OUTER JOIN'} += $n; $n = () = $str =~ m/\bCONNECT\s+BY\b/igs; $cost_details{'CONNECT BY'} += $n; $n = () = $str =~ m/\bBULK\s+COLLECT\b/igs; $cost_details{'BULK COLLECT'} += $n; $n = () = $str =~ m/\bFORALL\b/igs; $cost_details{'FORALL'} += $n; $n = () = $str =~ m/\bGOTO\b/igs; $cost_details{'GOTO'} += $n; $n = () = $str =~ m/\bROWNUM\b/igs; $cost_details{'ROWNUM'} += $n; $n = () = $str =~ m/\bNOTFOUND\b/igs; $cost_details{'NOTFOUND'} += $n; $n = () = $str =~ m/\bROWID\b/igs; $cost_details{'ROWID'} += $n; $n = () = $str =~ m/\bUROWID\b/igs; $cost_details{'UROWID'} += $n; $n = () = $str =~ m/\bSQLSTATE\b/igs; $cost_details{'SQLCODE'} += $n; $n = () = $str =~ m/\bIS RECORD\b/igs; $cost_details{'IS RECORD'} += $n; $n = () = $str =~ m/FROM[^;]*\bTABLE\s*\(/igs; $cost_details{'TABLE'} += $n; $n = () = $str =~ m/PIPE\s+ROW/igs; $cost_details{'PIPE ROW'} += $n; $n = () = $str =~ m/DBMS_\w/igs; $cost_details{'DBMS_'} += $n; $n = () = $str =~ m/DBMS_STANDARD\.RAISE EXCEPTION/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/UTL_\w/igs; $cost_details{'UTL_'} += $n; $n = () = $str =~ m/CTX_\w/igs; $cost_details{'CTX_'} += $n; $n = () = $str =~ m/\bEXTRACT\s*\(/igs; $cost_details{'EXTRACT'} += $n; $n = () = $str =~ m/\bTO_NUMBER\s*\(/igs; $cost_details{'TO_NUMBER'} += $n; # See: http://www.postgresql.org/docs/9.0/static/errcodes-appendix.html#ERRCODES-TABLE $n = () = $str =~ m/\b(DUP_VAL_ON_INDEX|TIMEOUT_ON_RESOURCE|TRANSACTION_BACKED_OUT|NOT_LOGGED_ON|LOGIN_DENIED|INVALID_NUMBER|PROGRAM_ERROR|VALUE_ERROR|ROWTYPE_MISMATCH|CURSOR_ALREADY_OPEN|ACCESS_INTO_NULL|COLLECTION_IS_NULL)\b/igs; $cost_details{'EXCEPTION'} += $n; $n = () = $str =~ m/PLUNIT/igs; $cost_details{'PLUNIT'} += $n; $n = () = $str =~ m/\bHT[PF]\./igs; $cost_details{'HTP'} += $n; if (!$class->{use_orafce}) { $n = () = $str =~ m/ADD_MONTHS/igs; $cost_details{'ADD_MONTHS'} += $n; $n = () = $str =~ m/LAST_DAY/igs; $cost_details{'LAST_DAY'} += $n; $n = () = $str =~ m/NEXT_DAY/igs; $cost_details{'NEXT_DAY'} += $n; $n = () = $str =~ m/MONTHS_BETWEEN/igs; $cost_details{'MONTHS_BETWEEN'} += $n; $n = () = $str =~ m/DBMS_OUTPUT\.put\(/igs; $cost_details{'DBMS_OUTPUT.put'} += $n; $n = () = $str =~ m/DBMS_OUTPUT\.(put_line|new_line|put)/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/\bTRUNC\s*\(/igs; $cost_details{'TRUNC'} += $n; $n = () = $str =~ m/REGEXP_LIKE/igs; $cost_details{'REGEXP_LIKE'} += $n; $n = () = $str =~ m/REGEXP_SUBSTR/igs; $cost_details{'REGEXP_SUBSTR'} += $n; $n = () = $str =~ m/REGEXP_COUNT/igs; $cost_details{'REGEXP_COUNT'} += $n; $n = () = $str =~ m/REGEXP_INSTR/igs; $cost_details{'REGEXP_INSTR'} += $n; $n = () = $str =~ m/PLVDATE/igs; $cost_details{'PLVDATE'} += $n; $n = () = $str =~ m/PLVSTR/igs; $cost_details{'PLVSTR'} += $n; $n = () = $str =~ m/PLVCHR/igs; $cost_details{'PLVCHR'} += $n; $n = () = $str =~ m/PLVSUBST/igs; $cost_details{'PLVSUBST'} += $n; $n = () = $str =~ m/PLVLEX/igs; $cost_details{'PLVLEX'} += $n; $n = () = $str =~ m/NVL2/igs; $cost_details{'NVL2'} += $n; } else { $n = () = $str =~ m/UTL_FILE/igs; $cost_details{'UTL_'} -= $n; $n = () = $str =~ m/DBMS_PIPE/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/DBMS_ALERT/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/DMS_UTILITY.FORMAT_CALL_STACK/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/DBMS_ASSERT/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/DBMS_STRING/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/PLUNIT.ASSERT/igs; $cost_details{'PLUNIT'} -= $n; $n = () = $str =~ m/DBMS_SQL/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/DBMS_RANDOM/igs; $cost_details{'DBMS_'} -= $n; $n = () = $str =~ m/NVL2/igs; $cost_details{'NVL2'} -= $n; } $n = () = $str =~ m/\b(INSERTING|DELETING|UPDATING)\b/igs; $cost_details{'TG_OP'} += $n; $n = () = $str =~ m/REF\s*CURSOR/igs; $cost_details{'CURSOR'} += $n; $n = () = $str =~ m/ORA_ROWSCN/igs; $cost_details{'ORA_ROWSCN'} += $n; $n = () = $str =~ m/SAVEPOINT/igs; $cost_details{'SAVEPOINT'} += $n; $n = () = $str =~ m/(FROM|EXEC)((?!WHERE).)*\b[\w\_]+\@[\w\_]+\b/igs; $cost_details{'DBLINK'} += $n; $n = () = $str =~ m/\%ISOPEN\b/igs; $cost_details{'ISOPEN'} += $n; $n = () = $str =~ m/\%ROWCOUNT\b/igs; $cost_details{'ROWCOUNT'} += $n; $str =~ s/MDSYS\.(["]*SDO_)/$1/igs; $n = () = $str =~ m/SDO_\w/igs; $cost_details{'SDO_'} += $n; $n = () = $str =~ m/PRAGMA/igs; $cost_details{'PRAGMA'} += $n; $n = () = $str =~ m/MDSYS\./igs; $cost_details{'MDSYS'} += $n; $n = () = $str =~ m/MERGE\sINTO/igs; $cost_details{'MERGE'} += $n; $n = () = $str =~ m/\bCONTAINS\(/igs; $cost_details{'CONTAINS'} += $n; $n = () = $str =~ m/\bSCORE\((?:.*)?\bCONTAINS\(/igs; $cost_details{'SCORE'} += $n; $n = () = $str =~ m/CONTAINS\((?:.*)?\bFUZZY\(/igs; $cost_details{'FUZZY'} += $n; $n = () = $str =~ m/CONTAINS\((?:.*)?\bNEAR\(/igs; $cost_details{'NEAR'} += $n; $n = () = $str =~ m/TO_CHAR\([^,\)]+\)/igs; $cost_details{'TO_CHAR'} += $n; $n = () = $str =~ m/TO_NCHAR\([^,\)]+\)/igs; $cost_details{'TO_NCHAR'} += $n; $n = () = $str =~ m/\s+ANYDATA/igs; $cost_details{'ANYDATA'} += $n; $n = () = $str =~ m/\|\|/igs; $cost_details{'CONCAT'} += $n; $n = () = $str =~ m/TIMEZONE_(REGION|ABBR)/igs; $cost_details{'TIMEZONE'} += $n; $n = () = $str =~ m/IS\s+(NOT)?\s*JSON/igs; $cost_details{'JSON'} += $n; $n = () = $str =~ m/TO_CLOB\([^,\)]+\)/igs; $cost_details{'TO_CLOB'} += $n; $n = () = $str =~ m/XMLTYPE\(/igs; $cost_details{'XMLTYPE'} += $n; $n = () = $str =~ m/CREATENONSCHEMABASEDXML\(/igs; $cost_details{'CREATENONSCHEMABASEDXML'} += $n; $n = () = $str =~ m/CREATESCHEMABASEDXML\(/igs; $cost_details{'CREATESCHEMABASEDXML'} += $n; $n = () = $str =~ m/CREATEXML\(/igs; $cost_details{'CREATEXML'} += $n; $n = () = $str =~ m/EXISTSNODE\(/igs; $cost_details{'EXISTSNODE'} += $n; $n = () = $str =~ m/EXTRACT\(/igs; $cost_details{'EXTRACT'} += $n; $n = () = $str =~ m/GETNAMESPACE\(/igs; $cost_details{'GETNAMESPACE'} += $n; $n = () = $str =~ m/GETROOTELEMENT\(/igs; $cost_details{'GETROOTELEMENT'} += $n; $n = () = $str =~ m/GETSCHEMAURL\(/igs; $cost_details{'GETSCHEMAURL'} += $n; $n = () = $str =~ m/ISFRAGMENT\(/igs; $cost_details{'ISFRAGMENT'} += $n; $n = () = $str =~ m/ISSCHEMABASED\(/igs; $cost_details{'ISSCHEMABASED'} += $n; $n = () = $str =~ m/ISSCHEMAVALID\(/igs; $cost_details{'ISSCHEMAVALID'} += $n; $n = () = $str =~ m/ISSCHEMAVALIDATED\(/igs; $cost_details{'ISSCHEMAVALIDATED'} += $n; $n = () = $str =~ m/SCHEMAVALIDATE\(/igs; $cost_details{'SCHEMAVALIDATE'} += $n; $n = () = $str =~ m/SETSCHEMAVALIDATED\(/igs; $cost_details{'SETSCHEMAVALIDATED'} += $n; $n = () = $str =~ m/TOOBJECT\(/igs; $cost_details{'TOOBJECT'} += $n; $n = () = $str =~ m/TRANSFORM\(/igs; $cost_details{'TRANSFORM'} += $n; $n = () = $str =~ m/ADD CONSTRAINT/igs; $cost_details{'ADD CONSTRAINT'} += $n; $n = () = $str =~ m/\bHT[PF]\./igs; $cost_details{'HTP'} += $n; $n = () = $str =~ m/,\s*'SSSSS'\s*\)/igs; $cost_details{"'SSSSS'"} += $n; $n = () = $str =~ m/,\s*'J'\s*\)/igs; $cost_details{"'J'"} += $n; $n = () = $str =~ m/WHEN\s+OTHER\s+THEN/igs; $cost_details{'WHEN OTHER'} += $n; foreach my $f (@ORA_FUNCTIONS) { if ($str =~ /\b$f\b/igs) { $cost += 1; $cost_details{$f} += 1; } } foreach my $t (keys %UNCOVERED_SCORE) { $cost += $UNCOVERED_SCORE{$t}*$cost_details{$t}; } return $cost, %cost_details; } =head2 mysql_to_plpgsql This function turn a MySQL function code into a PLPGSQL code =cut sub mysql_to_plpgsql { my ($class, $str) = @_; # remove FROM DUAL $str =~ s/FROM\s+DUAL//igs; # Simply remove this as not supported $str =~ s/\bDEFAULT\s+NULL\b//igs; # Change mysql variable affectation $str =~ s/(UPDATE\s+[^\s]+\s+SET\s+[^\s]+\s*)=/$1%EQUALSIGN%/igs; $str =~ s/\bSET\s+([^\s:=]+\s*)=([^;\n]+;)/$1:=$2/igs; $str =~ s/%EQUALSIGN%/=/igs; # replace simple form of json_extract $str =~ s/json_extract\(([^,]+),(?:.*)?('[^\\]+)\\/json_extract_path($1, $2/igs; $str =~ s/json_extract\(/json_extract_path(/igs; # remove declared handler $str =~ s/[^\s]+\s+HANDLER\s+FOR\s+[^;]+;//igs; # Fix call to unsigned $str =~ s/\bTINYINT\s+UNSIGNED\b/smallint/igs; $str =~ s/\bSMALLINT\s+UNSIGNED\b/integer/igs; $str =~ s/\bMEDIUMINT\s+UNSIGNED\b/integer/igs; $str =~ s/\bBIGINT\s+UNSIGNED\b/numeric/igs; $str =~ s/\bINT\s+UNSIGNED\b/bigint/igs; # Drop temporary doesn't exist in PostgreSQL $str =~ s/DROP\s+TEMPORARY/DROP/gs; # Private temporary table doesn't exist in PostgreSQL $str =~ s/PRIVATE\s+TEMPORARY/TEMPORARY/igs; $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; # Remove extra parenthesis in join in some possible cases # ... INNER JOIN(services s) ON ... $str =~ s/\bJOIN\s*\(([^\s]+\s+[^\s]+)\)/JOIN $1/igs; # Rewrite MySQL JOIN with WHERE clause instead of ON $str =~ s/\((\s*[^\s]+(?:\s+[^\s]+)?\s+JOIN\s+[^\s]+(?:\s+[^\s]+)?\s*)\)\s+WHERE\s+/$1 ON /igs; # Replace LEAVE by EXIT $str =~ s/\bLEAVE\s*;/EXIT;/igs; # Replace ITERATE by CONTINUE $str =~ s/\bITERATE\s*;/CONTINUE;/igs; # Replace now() with CURRENT_TIMESTAMP even if this is the same # because parenthesis can break the following regular expressions $str =~ s/\bNOW\(\s*\)/CURRENT_TIMESTAMP/igs; # Replace call to CURRENT_TIMESTAMP() to special variable $str =~ s/\bCURRENT_TIMESTAMP\s*\(\)/CURRENT_TIMESTAMP/igs; # Replace EXTRACT() with unit not supported by PostgreSQL if ($class->{mysql_internal_extract_format}) { $str =~ s/\bEXTRACT\(\s*YEAR_MONTH\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'YYYYMM')::integer/igs; $str =~ s/\bEXTRACT\(\s*DAY_HOUR\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24')::integer/igs; $str =~ s/\bEXTRACT\(\s*DAY_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MI')::integer/igs; $str =~ s/\bEXTRACT\(\s*DAY_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MISS')::integer/igs; $str =~ s/\bEXTRACT\(\s*DAY_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MISSUS')::bigint/igs; $str =~ s/\bEXTRACT\(\s*HOUR_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MI')::integer/igs; $str =~ s/\bEXTRACT\(\s*HOUR_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MISS')::integer/igs; $str =~ s/\bEXTRACT\(\s*HOUR_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MISSUS')::bigint/igs; $str =~ s/\bEXTRACT\(\s*MINUTE_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MISS')::integer/igs; $str =~ s/\bEXTRACT\(\s*MINUTE_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MISSUS')::bigint/igs; $str =~ s/\bEXTRACT\(\s*SECOND_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'SSUS')::integer/igs; } else { $str =~ s/\bEXTRACT\(\s*YEAR_MONTH\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'YYYY-MM')/igs; $str =~ s/\bEXTRACT\(\s*DAY_HOUR\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24')/igs; $str =~ s/\bEXTRACT\(\s*DAY_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI')/igs; $str =~ s/\bEXTRACT\(\s*DAY_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI:SS')/igs; $str =~ s/\bEXTRACT\(\s*DAY_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI:SS.US')/igs; $str =~ s/\bEXTRACT\(\s*HOUR_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI')/igs; $str =~ s/\bEXTRACT\(\s*HOUR_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI:SS')/igs; $str =~ s/\bEXTRACT\(\s*HOUR_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI:SS.US')/igs; $str =~ s/\bEXTRACT\(\s*MINUTE_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MI:SS')/igs; $str =~ s/\bEXTRACT\(\s*MINUTE_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MI:SS.US')/igs; $str =~ s/\bEXTRACT\(\s*SECOND_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'SS.US')/igs; } # Replace operators if (!$class->{mysql_pipes_as_concat}) { $str =~ s/\|\|/ OR /igs; $str =~ s/\&\&/ AND /igs; } $str =~ s/BIT_XOR\(\s*([^,]+)\s*,\s*(\d+)\s*\)/$1 # coalesce($2, 0)/igs; $str =~ s/\bXOR\b/#/igs; $str =~ s/\b\^\b/#/igs; #### # Replace some function with their PostgreSQL syntax #### # Math related fucntion $str =~ s/\bATAN\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/atan2($1, $2)/igs; $str =~ s/\bLOG\(/ln\(/igs; $str =~ s/\bLOG10\(\s*([^\(\)]+)\s*\)/log\(10, $1\)/igs; $str =~ s/\bLOG2\(\s*([^\(\)]+)\s*\)/log\(2, $1\)/igs; $str =~ s/([^\(\s]+)\s+MOD\s+([^\s\)]+)/mod\($1, $2\)/igs; $str =~ s/\bPOW\(/power\(/igs; $str =~ s/\bRAND\(\s*\)/random\(\)/igs; # Misc function $str =~ s/\bCHARSET\(\s*([^\(\)]+)\s*\)/current_setting('server_encoding')/igs; $str =~ s/\bCOLLATION\(\s*([^\(\)]+)\s*\)/current_setting('lc_collate')/igs; $str =~ s/\bCONNECTION_ID\(\s*\)/pg_backend_pid()/igs; $str =~ s/\b(DATABASE|SCHEMA)\(\s*\)/current_database()/igs; $str =~ s/\bSLEEP\(/pg_sleep\(/igs; $str =~ s/\bSYSTEM_USER\(\s*\)/CURRENT_USER/igs; $str =~ s/\bSESSION_USER\(\s*\)/SESSION_USER/igs; $str =~ s/\bTRUNCATE\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/trunc\($1, $2\)/igs; $str =~ s/\bUSER\(\s*\)/CURRENT_USER/igs; # Date/time related function $str =~ s/\b(CURDATE|CURRENT_DATE)\(\s*\)/CURRENT_DATE/igs; $str =~ s/\b(CURTIME|CURRENT_TIME)\(\s*\)/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; $str =~ s/\bCURRENT_TIMESTAMP\(\s*\)/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; $str =~ s/\b(LOCALTIMESTAMP|LOCALTIME)\(\s*\)/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; $str =~ s/\b(LOCALTIMESTAMP|LOCALTIME)\b/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; $str =~ s/\bstatementSYSDATE\(\s*\)/timeofday()::timestamp(0) without time zone/igs; $str =~ s/\bUNIX_TIMESTAMP\(\s*\)/floor(extract(epoch from CURRENT_TIMESTAMP::timestamp with time zone))/igs; $str =~ s/\bUNIX_TIMESTAMP\(\s*([^\)]+)\s*\)/floor(extract(epoch from ($1)::timestamp with time zone))/igs; $str =~ s/\bUTC_DATE\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::date/igs; $str =~ s/\bUTC_TIME\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::time(0)/igs; $str =~ s/\bUTC_TIMESTAMP\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::timestamp(0)/igs; $str =~ s/\bCONVERT_TZ\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/(($1)::timestamp without time zone AT TIME ZONE ($2)::text) AT TIME ZONE ($3)::text/igs; $str =~ s/\bDATEDIFF\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/extract(day from (date_trunc('day', ($1)::timestamp) - date_trunc('day', ($2)::timestamp)))/igs; $str =~ s/\bDATE_FORMAT\(\s*(.*?)\s*,\s*('[^'\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_dateformat_to_pgsql($class, $1, $2)/iges; $str =~ s/\b(?:ADDDATE|DATE_ADD)\(\s*(.*?)\s*,\s*INTERVAL\s*([^\(\),]+)\s*\)/"($1)::timestamp " . _replace_dateadd($2)/iges; $str =~ s/\bADDDATE\(\s*([^,]+)\s*,\s*(\d+)\s*\)/($1)::timestamp + ($2 * interval '1 day')/igs; $str =~ s/\bADDTIME\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp + ($2)::interval/igs; $str =~ s/\b(DAY|DAYOFMONTH)\(\s*([^\(\)]+)\s*\)/extract(day from date($1))::integer/igs; $str =~ s/\bDAYNAME\(\s*([^\(\)]+)\s*\)/to_char(($1)::date, 'FMDay')/igs; $str =~ s/\bDAYOFWEEK\(\s*([^\(\)]+)\s*\)/extract(dow from date($1))::integer + 1/igs; # start on sunday = 1 $str =~ s/\bDAYOFYEAR\(\s*([^\(\)]+)\s*\)/extract(doy from date($1))::integer/igs; $str =~ s/\bFORMAT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/to_char(round($1, $2), 'FM999,999,999,999,999,999,990'||case when $2 > 0 then '.'||repeat('0', $2) else '' end)/igs; $str =~ s/\bFROM_DAYS\(\s*([^\(\)]+)\s*\)/'0001-01-01bc'::date + ($1)::integer/igs; $str =~ s/\bFROM_UNIXTIME\(\s*([^\(\),]+)\s*\)/to_timestamp($1)::timestamp without time zone/igs; $str =~ s/\bFROM_UNIXTIME\(\s*(.*?)\s*,\s*('[^\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/FROM_UNIXTIME2(to_timestamp($1), $2)/igs; $str =~ s/\bFROM_UNIXTIME2\(\s*(.*?)\s*,\s*('[^'\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_dateformat_to_pgsql($class, $1, $2)/eigs; $str =~ s/\bGET_FORMAT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/_mysql_getformat_to_pgsql($1, $2)/eigs; $str =~ s/\bHOUR\(\s*([^\(\)]+)\s*\)/extract(hour from ($1)::interval)::integer/igs; $str =~ s/\bLAST_DAY\(\s*([^\(\)]+)\s*\)/((date_trunc('month',($1)::timestamp + interval '1 month'))::date - 1)/igs; $str =~ s/\bMAKEDATE\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(date($1||'-01-01') + ($2 - 1) * interval '1 day')::date/igs; $str =~ s/\bMAKETIME\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1 * interval '1 hour' + $2 * interval '1 min' + $3 * interval '1 sec')/igs; $str =~ s/\bMICROSECOND\(\s*([^\(\)]+)\s*\)/extract(microsecond from ($1)::time)::integer/igs; $str =~ s/\bMINUTE\(\s*([^\(\)]+)\s*\)/extract(minute from ($1)::time)::integer/igs; $str =~ s/\bMONTH\(\s*([^\(\)]+)\s*\)/extract(month from date($1))::integer/igs; $str =~ s/\bMONTHNAME\(\s*([^\(\)]+)\s*\)/to_char(($1)::date, 'FMMonth')/igs; $str =~ s/\bQUARTER\(\s*([^\(\)]+)\s*\)/extract(quarter from date($1))::integer/igs; $str =~ s/\bSECOND\(\s*([^\(\)]+)\s*\)/extract(second from ($1)::interval)::integer/igs; $str =~ s/\bSEC_TO_TIME\(\s*([^\(\)]+)\s*\)/($1 * interval '1 second')/igs; $str =~ s/\bSTR_TO_DATE\(\s*(.*?)\s*,\s*('[^'\(\),]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_strtodate_to_pgsql($class, $1, $2)/eigs; $str =~ s/\b(SUBDATE|DATE_SUB)\(\s*([^,]+)\s*,\s*INTERVAL ([^\(\)]+)\s*\)/($2)::timestamp - interval '$3'/igs; $str =~ s/\bSUBDATE\(\s*([^,]+)\s*,\s*(\d+)\s*\)/($1)::timestamp - ($2 * interval '1 day')/igs; $str =~ s/\bSUBTIME\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp - ($2)::interval/igs; $str =~ s/\bTIME(\([^\(\)]+\))/($1)::time/igs; $str =~ s/\bTIMEDIFF\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp - ($2)::timestamp/igs; #$str =~ s/\bTIMESTAMP\(\s*([^\(\)]+)\s*\)/($1)::timestamp/igs; $str =~ s/\bTIMESTAMP\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp + ($2)::time/igs; $str =~ s/\bTIMESTAMPADD\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($3)::timestamp + ($1 * interval '1 $2')/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*YEAR\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(year from ($2)::timestamp) - extract(year from ($1)::timestamp)/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*MONTH\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/(extract(year from ($2)::timestamp) - extract(year from ($1)::timestamp))*12 + (extract(month from ($2)::timestamp) - extract(month from ($1)::timestamp))/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*WEEK\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(day from ( ($2)::timestamp - ($1)::timestamp))\/7)/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*DAY\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(day from ( ($2)::timestamp - ($1)::timestamp))/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*HOUR\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(epoch from ( ($2)::timestamp - ($1)::timestamp))\/3600)/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*MINUTE\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(epoch from ( ($2)::timestamp - ($1)::timestamp))\/60)/igs; $str =~ s/\bTIMESTAMPDIFF\(\s*SECOND\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(epoch from ($2)::timestamp) - extract(epoch from ($1)::timestamp))/igs; $str =~ s/\bTIME_FORMAT\(\s*(.*?)\s*,\s*('[^'\(\),]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_timeformat_to_pgsql($class, $1, $2)/eigs; $str =~ s/\bTIME_TO_SEC\(\s*([^\(\)]+)\s*\)/(extract(hours from ($1)::time)*3600 + extract(minutes from ($1)::time)*60 + extract(seconds from ($1)::time))::bigint/igs; $str =~ s/\bTO_DAYS\(\s*([^\(\)]+)\s*\)/(($1)::date - '0001-01-01bc')::integer/igs; $str =~ s/\bWEEK(\([^\(\)]+\))/extract(week from date($1)) - 1/igs; $str =~ s/\bWEEKOFYEAR(\([^\(\)]+\))/extract(week from date($2))/igs; $str =~ s/\bWEEKDAY\(\s*([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'ID')::integer - 1/igs; # MySQL: Monday = 0, PG => 1 $str =~ s/\bYEAR\(\s*([^\(\)]+)\s*\)/extract(year from date($1))/igs; # String functions $str =~ s/\bBIN\(\s*([^\(\)]+)\s*\)/ltrim(textin(bit_out($1::bit(64))), '0')/igs; $str =~ s/\bBINARY\(\s*([^\(\)]+)\s*\)/($1)::bytea/igs; $str =~ s/\bBIT_COUNT\(\s*([^\(\)]+)\s*\)/length(replace(ltrim(textin(bit_out($1::bit(64))),'0'),'0',''))/igs; $str =~ s/\bCHAR\(\s*([^\(\),]+)\s*\)/array_to_string(ARRAY(SELECT chr(unnest($1))),'')/igs; $str =~ s/\bELT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(ARRAY[$2])[$1]/igs; $str =~ s/\bFIELD\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(SELECT i FROM generate_subscripts(array[$2], 1) g(i) WHERE $1 = (array[$2])[i] UNION ALL SELECT 0 LIMIT 1)/igs; $str =~ s/\bFIND_IN_SET\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(SELECT i FROM generate_subscripts(string_to_array($2,','), 1) g(i) WHERE $1 = (string_to_array($2,','))[i] UNION ALL SELECT 0 LIMIT 1)/igs; $str =~ s/\bFROM_BASE64\(\s*([^\(\),]+)\s*\)/decode(($1)::bytea, 'base64')/igs; $str =~ s/\bHEX\(\s*([^\(\),]+)\s*\)/upper(encode($1::bytea, 'hex'))/igs; $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*('[^']+')\s*\)/position($2 in $1)/igs; if (!$class->{pg_supports_substr}) { $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in substring ($2 from $3)) + $3 - 1/igs; $str =~ s/\bMID\(/substring\(/igs; } else { $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in substr($2, $3)) + $3 - 1/igs; $str =~ s/\bMID\(/substr\(/igs; } $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in $2)/igs; $str =~ s/\bLCASE\(/lower\(/igs; $str =~ s/\bORD\(/ascii\(/igs; $str =~ s/\bQUOTE\(/quote_literal\(/igs; $str =~ s/\bSPACE\(\s*([^\(\),]+)\s*\)/repeat(' ', $1)/igs; $str =~ s/\bSTRCMP\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/CASE WHEN $1 < $2 THEN -1 WHEN $1 > $2 THEN 1 ELSE 0 END/igs; $str =~ s/\bTO_BASE64\(\s*([^\(\),]+)\s*\)/encode($1, 'base64')/igs; $str =~ s/\bUCASE\(/upper\(/igs; $str =~ s/\bUNHEX\(\s*([^\(\),]+)\s*\)/decode($1, 'hex')::text/igs; $str =~ s/\bIS_IPV6\(\s*([^\(\)]+)\s*\)/CASE WHEN family($1) = 6 THEN 1 ELSE 0 END/igs; $str =~ s/\bIS_IPV4\(\s*([^\(\)]+)\s*\)/CASE WHEN family($1) = 4 THEN 1 ELSE 0 END/igs; $str =~ s/\bISNULL\(\s*([^\(\)]+)\s*\)/$1 IS NULL/igs; $str =~ s/\bRLIKE/REGEXP/igs; $str =~ s/\bSTD\(/STDDEV_POP\(/igs; $str =~ s/\bSTDDEV\(/STDDEV_POP\(/igs; $str =~ s/\bUUID\(/$class->{uuid_function}\(/igs; $str =~ s/\bNOT REGEXP BINARY/\!\~/igs; $str =~ s/\bREGEXP BINARY/\~/igs; $str =~ s/\bNOT REGEXP/\!\~\*/igs; $str =~ s/\bREGEXP/\~\*/igs; $str =~ s/\bGET_LOCK/pg_advisory_lock/igs; $str =~ s/\bIS_USED_LOCK/pg_try_advisory_lock/igs; $str =~ s/\bRELEASE_LOCK/pg_advisory_unlock/igs; # GROUP_CONCAT doesn't exist, it must be replaced by calls to array_to_string() and array_agg() functions $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+(ASC|DESC)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1 ORDER BY $2 $3), $4)/igs; $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1 ORDER BY $2 ASC), $3)/igs; $str =~ s/GROUP_CONCAT\((.*?)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1), $2)/igs; $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+(ASC|DESC)\s*\)/array_to_string(array_agg($1 ORDER BY $2 $3), ',')/igs; $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s*\)/array_to_string(array_agg($1 ORDER BY $2), ',')/igs; $str =~ s/GROUP_CONCAT\(([^\)]+)\)/array_to_string(array_agg($1), ',')/igs; # Replace IFNULL() MySQL function in a query while ($str =~ s/\bIFNULL\(\s*([^,]+)\s*,\s*([^\)]+\s*)\)/COALESCE($1, $2)/is) {}; # Rewrite while loop $str =~ s/\bWHILE\b(.*?)\bEND\s+WHILE\s*;/WHILE $1END LOOP;/igs; $str =~ s/\bWHILE\b(.*?)\bDO\b/WHILE $1LOOP/igs; # Rewrite REPEAT loop my %repl_repeat = (); $i = 0; while ($str =~ s/\bREPEAT\s+(.*?)\bEND REPEAT\s*;/%REPREPEATLBL$i%/igs) { my $code = $1; $code =~ s/\bUNTIL(.*)//; $repl_repeat{$i} = "LOOP ${code}EXIT WHEN $1;\nEND LOOP;"; } foreach $i (keys %repl_repeat) { $str =~ s/\%REPREPEATLBL$i\%/$repl_repeat{$i}/gs; } %repl_repeat = (); # Fix some charset encoding call in cast function #$str =~ s/(CAST\s*\((?:.*?)\s+AS\s+(?:[^\s]+)\s+)CHARSET\s+([^\s\)]+)\)/$1) COLLATE "\U$2\E"/igs; $str =~ s/(CAST\s*\((?:.*?)\s+AS\s+(?:[^\s]+)\s+)(CHARSET|CHARACTER\s+SET)\s+([^\s\)]+)\)/$1)/igs; $str =~ s/CONVERT\s*(\((?:[^,]+)\s+,\s+(?:[^\s]+)\s+)(CHARSET|CHARACTER\s+SET)\s+([^\s\)]+)\)/CAST$1)/igs; $str =~ s/CONVERT\s*\((.*?)\s+USING\s+([^\s\)]+)\)/CAST($1 AS text)/igs; # Set default UTF8 collation to postgreSQL equivalent C.UTF-8 #$str =~ s/COLLATE "UTF8"/COLLATE "C.UTF-8"/gs; $str =~ s/\bCHARSET(\s+)/COLLATE$1/igs; # Remove call to start transaction $str =~ s/\sSTART\s+TRANSACTION\s*;/-- START TRANSACTION;/igs; # Comment call to COMMIT or ROLLBACK in the code if allowed if ($class->{comment_commit_rollback}) { $str =~ s/\b(COMMIT|ROLLBACK)\s*;/-- $1;/igs; $str =~ s/(ROLLBACK\s+TO\s+[^;]+);/-- $1;/igs; } # Translate call to CREATE TABLE ... SELECT $str =~ s/CREATE\s+PRIVATE\s+TEMPORARY/CREATE TEMPORARY/; $str =~ s/(CREATE(?:\s+TEMPORARY)?\s+TABLE\s+[^\s]+)(\s+SELECT)/$1 AS $2/igs; $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; # Remove @ from variables and rewrite SET assignement in QUERY mode if ($class->{type} eq 'QUERY') { $str =~ s/\@([^\s]+)\b/$1/gs; $str =~ s/:=/=/gs; } # Replace spatial related lines $str = replace_mysql_spatial($str); # Rewrite direct call to function without out parameters using PERFORM $str = perform_replacement($class, $str); # Remove CALL from all statements if not supported if (!$class->{pg_supports_procedure}) { $str =~ s/\bCALL\s+//igs; } if ($str =~ /\s+FROM\s+(.*?)\s+WHERE/is) { my @joins = split(/\bJOIN\b/i, $1); my $res = ''; foreach my $j (@joins) { if ($res eq '') { $res = $j; } elsif ($j !~ /\bON\b/i) { $res .= ',' . $j; } else { $res .= 'JOIN' . $j; } } $str =~ s/(\s+FROM\s+)(.*?)(\s+WHERE)/$1$res$3/is; } return $str; } sub _replace_dateadd { my $str = shift; my $dd = shift; my $op = '+'; if ($str =~ s/^\-[\s]*//) { $op = '-'; } if ($str =~ s/^(\d+)\s+([^\(\),\s]+)$/ $op $1*interval '1 $2'/s) { return $str; } elsif ($str =~ s/^([^\s]+)\s+([^\(\),\s]+)$/ $op $1*interval '1 $2'/s) { return $str; } elsif ($str =~ s/^([^\(\),]+)$/ $op interval '$1'/s) { return $str; } return $str; } sub replace_mysql_spatial { my $str = shift; $str =~ s/AsWKB\(/AsBinary\(/igs; $str =~ s/AsWKT\(/AsText\(/igs; $str =~ s/GeometryCollectionFromText\(/GeomCollFromText\(/igs; $str =~ s/GeometryCollectionFromWKB\(/GeomCollFromWKB\(/igs; $str =~ s/GeometryFromText\(/GeomFromText\(/igs; $str =~ s/GLength\(/ST_Length\(/igs; $str =~ s/LineStringFromWKB\(/LineFromWKB\(/igs; $str =~ s/MultiLineStringFromText\(/MLineFromText\(/igs; $str =~ s/MultiPointFromText\(/MPointFromText\(/igs; $str =~ s/MultiPolygonFromText\(/MPolyFromText\(/igs; $str =~ s/PolyFromText\(/PolygonFromText\(/igs; $str =~ s/MBRContains\(/ST_Contains\(/igs; $str =~ s/MBRDisjoint\(/ST_Disjoint\(/igs; $str =~ s/MBREqual\(/ST_Equals\(/igs; $str =~ s/MBRIntersects\(/ST_Intersects\(/igs; $str =~ s/MBROverlaps\(/ST_Overlaps\(/igs; $str =~ s/MBRTouches\(/ST_Touches\(/igs; $str =~ s/MBRWithin\(/ST_Within\(/igs; $str =~ s/MLineFromWKB\(/MultiLineStringFromWKB\(/igs; $str =~ s/MPointFromWKB\(/MultiPointFromWKB\(/igs; $str =~ s/MPolyFromWKB\(/MultiPolygonFromWKB\(/igs; $str =~ s/PolyFromWKB\(/PolygonFromWKB\(/igs; # Replace FromWKB functions foreach my $fct ('MultiLineStringFromWKB', 'MultiPointFromWKB', 'MultiPolygonFromWKB', 'PolygonFromWKB') { $str =~ s/\b$fct\(/ST_GeomFromWKB\(/igs; } # Add ST_ prefix to function alias foreach my $fct (@MYSQL_SPATIAL_FCT) { $str =~ s/\b$fct\(/ST_$fct\(/igs; } return $str; } sub _mysql_getformat_to_pgsql { my ($type, $format) = @_; if (uc($type) eq 'DATE') { if (uc($format) eq "'USA'") { $format = "'%m.%d.%Y'"; } elsif (uc($format) eq "'EUR'") { $format = "'%d.%m.%Y'"; } elsif (uc($format) eq "'INTERNAL'") { $format = "'%Y%m%d'"; } else { # ISO and JIS $format = "'%Y-%m-%d'"; } } elsif (uc($type) eq 'TIME') { if (uc($format) eq "'USA'") { $format = "'%h:%i:%s %p'"; } elsif (uc($format) eq "'EUR'") { $format = "'%H.%i.%s'"; } elsif (uc($format) eq "'INTERNAL'") { $format = "'%H%i%s'"; } else { # ISO and JIS $format = "'%H:%i:%s'"; } } else { if ( (uc($format) eq "'USA'") || (uc($format) eq "'EUR'") ) { $format = "'%Y-%m-%d %H.%i.%s'"; } elsif (uc($format) eq "'INTERNAL'") { $format = "'%Y%m%d%H%i%s'"; } else { # ISO and JIS $format = "'%Y-%m-%d %H:%i:%s'"; } } return $format; } sub _mysql_strtodate_to_pgsql { my ($class, $datetime, $format) = @_; my $str = _mysql_dateformat_to_pgsql($class, $datetime, $format, 1); return $str; } sub _mysql_timeformat_to_pgsql { my ($class, $datetime, $format) = @_; my $str = _mysql_dateformat_to_pgsql($class, $datetime, $format, 0, 1); return $str; } sub _mysql_dateformat_to_pgsql { my ($class, $datetime, $format, $todate, $totime) = @_; # Not supported: # %X Year for the week where Sunday is the first day of the week, numeric, four digits; used with %V $format =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/igs; $format =~ s/\%a/Dy/g; $format =~ s/\%b/Mon/g; $format =~ s/\%c/FMMM/g; $format =~ s/\%D/FMDDth/g; $format =~ s/\%e/FMDD/g; $format =~ s/\%f/US/g; $format =~ s/\%H/HH24/g; $format =~ s/\%h/HH12/g; $format =~ s/\%I/HH/g; $format =~ s/\%i/MI/g; $format =~ s/\%j/DDD/g; $format =~ s/\%k/FMHH24/g; $format =~ s/\%l/FMHH12/g; $format =~ s/\%m/MM/g; $format =~ s/\%p/AM/g; $format =~ s/\%r/HH12:MI:SS AM/g; $format =~ s/\%s/SS/g; $format =~ s/\%S/SS/g; $format =~ s/\%T/HH24:MI:SS/g; $format =~ s/\%U/WW/g; $format =~ s/\%u/IW/g; $format =~ s/\%V/WW/g; $format =~ s/\%v/IW/g; $format =~ s/\%x/YYYY/g; $format =~ s/\%X/YYYY/g; $format =~ s/\%Y/YYYY/g; $format =~ s/\%y/YY/g; $format =~ s/\%W/Day/g; $format =~ s/\%M/Month/g; $format =~ s/\%(\d+)/$1/g; # Replace constant strings if ($format =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/is) { $class->{text_values}{$class->{text_values_pos}} = $1; $class->{text_values_pos}++; } if ($todate) { return "to_date($datetime, $format)"; } elsif ($totime) { return "to_char(($datetime)::time, $format)"; } return "to_char(($datetime)::timestamp, $format)"; } sub mysql_estimate_cost { my ($class, $str, $type) = @_; my %cost_details = (); # Default cost is testing that mean it at least must be tested my $cost = $FCT_TEST_SCORE; # When evaluating queries tests must not be included here if ($type eq 'QUERY') { $cost = 0; } $cost_details{'TEST'} = $cost; # Set cost following code length my $cost_size = int(length($str)/$SIZE_SCORE) || 1; # When evaluating queries size must not be included here if ($type eq 'QUERY') { $cost_size = 0; } $cost += $cost_size; $cost_details{'SIZE'} = $cost_size; # Try to figure out the manual work my $n = () = $str =~ m/(ARRAY_AGG|GROUP_CONCAT)\(\s*DISTINCT/igs; $cost_details{'ARRAY_AGG_DISTINCT'} += $n; $n = () = $str =~ m/\bSOUNDS\s+LIKE\b/igs; $cost_details{'SOUNDS LIKE'} += $n; $n = () = $str =~ m/CHARACTER\s+SET/igs; $cost_details{'CHARACTER SET'} += $n; $n = () = $str =~ m/\bCOUNT\(\s*DISTINCT\b/igs; $cost_details{'COUNT(DISTINCT)'} += $n; $n = () = $str =~ m/\bMATCH.*AGAINST\b/igs; $cost_details{'MATCH'} += $n; $n = () = $str =~ m/\bJSON_[A-Z\_]+\(/igs; $cost_details{'JSON FUNCTION'} += $n; $n = () = $str =~ m/_(un)?lock\(/igs; $cost_details{'LOCK'} += $n; $n = () = $str =~ m/\b\@+[A-Z0-9\_]+/igs; $cost_details{'@VAR'} += $n; foreach my $t (keys %UNCOVERED_MYSQL_SCORE) { $cost += $UNCOVERED_MYSQL_SCORE{$t}*$cost_details{$t}; } foreach my $f (@MYSQL_FUNCTIONS) { if ($str =~ /\b$f\b/igs) { $cost += 2; $cost_details{$f} += 2; } } return $cost, %cost_details; } sub mssql_estimate_cost { my ($class, $str, $type) = @_; my %cost_details = (); # TSQL do not use ; as statements separator and condition use begin instead of then/loop... # this require manual editing so decrease the number of lines for cost of the code review. $SIZE_SCORE = 400; # Default cost is testing that mean it at least must be tested my $cost = $FCT_TEST_SCORE; # When evaluating queries tests must not be included here if ($type eq 'QUERY') { $cost = 0; } $cost_details{'TEST'} = $cost; # Set cost following code length my $cost_size = int(length($str)/$SIZE_SCORE) || 1; # When evaluating queries size must not be included here if ($type eq 'QUERY') { $cost_size = 0; } $cost += $cost_size; $cost_details{'SIZE'} = $cost_size; # Try to figure out the manual work # Not accurate for now my $n = () = $str =~ m/(ARRAY_AGG|GROUP_CONCAT)\(\s*DISTINCT/igs; $cost_details{'ARRAY_AGG_DISTINCT'} += $n*$UNCOVERED_MSSQL_SCORE{'ARRAY_AGG_DISTINCT'}; # Look for access to objects in other database, require FDW or dblink. $n = () = $str =~ /\b[a-z0-9_\$]+\.[a-z0-9_\$]+\.[a-z0-9_\$]+\b/igs; $cost_details{'FOREIGN_OBJECT'} += $n*$UNCOVERED_MSSQL_SCORE{'FOREIGN_OBJECT'}; $n = () = $str =~ /\b(master|model|msdb|tempdb)\.\b/igs; $cost_details{'SYS_OBJECT'} += $n*$UNCOVERED_MSSQL_SCORE{'SYS_OBJECT'}; $n = () = $str =~ /\bOBJECT_ID\s*\(/igs; $cost_details{'OBJECT_ID'} += $n*$UNCOVERED_MSSQL_SCORE{'OBJECT_ID'}; #$cost_details{'FOREIGN_OBJECT'} -= $n*$UNCOVERED_MSSQL_SCORE{'FOREIGN_OBJECT'}; if ($class->{local_schemas_regex}) { $n = () = $str =~ /\b$class->{local_schemas_regex}\.[a-z0-9_\$]+\.[a-z0-9_\$]+\b/igs; $cost_details{'FOREIGN_OBJECT'} -= $n*$UNCOVERED_MSSQL_SCORE{'FOREIGN_OBJECT'}; } $n = () = $str =~ /[\s,]\s*sys[a-z]+/igs; $cost_details{'SYS_OBJECT'} += $n*$UNCOVERED_MSSQL_SCORE{'SYS_OBJECT'}; $n = () = $str =~ /\b\#\#[a-z0-9_\$]+\b/igs; $cost_details{'GLOBAL_TEMP_TABLE'} += $n*$UNCOVERED_MSSQL_SCORE{'GLOBAL_TEMP_TABLE'}; $n = () = $str =~ /(?{use_mssqlfce} && $f =~ /^(DATEDIFF|STUFF|PATINDEX|ISNUMERIC|ISDATE|LEN|PRINT)$/); if ($str =~ /\b$f\s*\(/igs) { $cost += 2; $cost_details{$f} += 2; } } return $cost, %cost_details; } sub replace_outer_join { my ($class, $str, $type) = @_; # Remove comments in the from clause. They need to be removed because the # entire FROM clause will be rewritten and we don't know where to restore. while ($str =~ s/(\s+FROM\s+(?:.*)?)\%ORA2PG_COMMENT\d+\%((?:.*)?WHERE\s+)/$1$2/is) {}; if (!grep(/^$type$/, 'left', 'right')) { die "FATAL: outer join type must be 'left' or 'right' in call to replace_outer_join().\n"; } # When we have a right outer join, just rewrite it as a left join to simplify the translation work if ($type eq 'right') { $str =~ s/(\s+)([^\s]+)\s*(\%OUTERJOIN\d+\%)\s*(!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)\s*([^\s]+)/$1$5 $4 $2$3/isg; return $str; } my $regexp1 = qr/((?:!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)\s*[^\s]+\s*\%OUTERJOIN\d+\%)/is; my $regexp2 = qr/\%OUTERJOIN\d+\%\s*(?:!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)/is; # process simple form of outer join my $nbouter = $str =~ $regexp1; # Check that we don't have right outer join too if ($nbouter >= 1 && $str !~ $regexp2) { # Extract tables in the FROM clause $str =~ s/(.*)\bFROM\s+(.*?)\s+WHERE\s+(.*?)$/$1FROM FROM_CLAUSE WHERE $3/is; my $from_clause = $2; $from_clause =~ s/"//gs; my @tables = split(/\s*,\s*/, $from_clause); # Set a hash for alias to table mapping my %from_clause_list = (); my %from_order = (); my $fidx = 0; foreach my $table (@tables) { $table =~ s/^\s+//s; $table =~ s/\s+$//s; my $cmt = ''; while ($table =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { $cmt .= $1; } my ($t, $alias, @others) = split(/\s+/, lc($table)); $alias = $others[0] if (uc($alias) eq 'AS'); $alias = "$t" if (!$alias); $from_clause_list{$alias} = "$cmt$t"; $from_order{$alias} = $fidx++; } # Extract all Oracle's outer join syntax from the where clause my @outer_clauses = (); my %final_outer_clauses = (); my %final_from_clause = (); my @tmp_from_list = (); my $start_query = ''; my $end_query = ''; if ($str =~ s/^(.*FROM FROM_CLAUSE WHERE)//is) { $start_query = $1; } if ($str =~ s/\s+((?:START WITH|CONNECT BY|ORDER SIBLINGS BY|GROUP BY|ORDER BY).*)$//is) { $end_query = $1; } # Extract predicat from the WHERE clause my @predicat = split(/\s*(\bAND\b|\bOR\b|\%ORA2PG_COMMENT\d+\%)\s*/i, $str); my $id = 0; my %other_join_clause = (); # Process only predicat with a obsolete join syntax (+) for now for (my $i = 0; $i <= $#predicat; $i++) { next if ($predicat[$i] !~ /\%OUTERJOIN\d+\%/i); # remove extrat parenthesis from the predicat for better parsing $predicat[$i] =~ s/^\s*[\(]*\s*([^\(\)]+)\s*[\)]*\s*/$1/gs; my $where_clause = $predicat[$i]; $where_clause =~ s/"//gs; $where_clause =~ s/^\s+//s; $where_clause =~ s/[\s;]+$//s; $where_clause =~ s/\s*(\%OUTERJOIN\d+\%)//gs; $predicat[$i] = "WHERE_CLAUSE$id "; # Split the predicat to retrieve left part, operator and right part my ($l, $o, $r) = split(/\s*(!=|>=|<=|=|<>|<|>|NOT LIKE|LIKE)\s*/i, $where_clause); # NEW / OLD pseudo table in triggers can not be part of a join # clause. Move them int to the WHERE clause. if ($l =~ /^(NEW|OLD)\./is) { $predicat[$i] =~ s/WHERE_CLAUSE$id / $l $o $r /s; next; } $id++; # Extract the tablename part of the left clause my $lbl1 = ''; my $table_decl1 = $l; if ($l =~ /^([^\.\s]+\.[^\.\s]+)\..*/ || $l =~ /^([^\.\s]+)\..*/) { $lbl1 = lc($1); $lbl1 =~ s/\(\s*//; # If the table/alias is not part of the from clause if (!exists $from_clause_list{$lbl1}) { $from_clause_list{$lbl1} = $lbl1; $from_order{$lbl1} = $fidx++; } $table_decl1 = $from_clause_list{$lbl1}; $table_decl1 .= " $lbl1" if ($lbl1 ne $from_clause_list{$lbl1}); } elsif ($l =~ /\%SUBQUERY(\d+)\%/) { # Search for table.column in the subquery or function code my $tmp_str = $l; while ($tmp_str =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { if ($tmp_str =~ /\b([^\.\s]+\.[^\.\s]+)\.[^\.\s]+/ || $tmp_str =~ /\b([^\.\s]+)\.[^\.\s]+/) { $lbl1 = lc($1); $lbl1 =~ s/\(\s*//; # If the table/alias is not part of the from clause if (!exists $from_clause_list{$lbl1}) { $from_clause_list{$lbl1} = $lbl1; $from_order{$lbl1} = $fidx++; } $table_decl1 = $from_clause_list{$lbl1}; $table_decl1 .= " $lbl1" if ($lbl1 ne $from_clause_list{$lbl1}); last; } } } # Extract the tablename part of the right clause my $lbl2 = ''; my $table_decl2 = $r; if ($r =~ /^([^\.\s]+\.[^\.\s]+)\..*/ || $r =~ /^([^\.\s]+)\..*/) { $lbl2 = lc($1); if (!$lbl1) { push(@{$other_join_clause{$lbl2}}, "$l $o $r"); next; } # If the table/alias is not part of the from clause if (!exists $from_clause_list{$lbl2}) { $from_clause_list{$lbl2} = $lbl2; $from_order{$lbl2} = $fidx++; } $table_decl2 = $from_clause_list{$lbl2}; $table_decl2 .= " $lbl2" if ($lbl2 ne $from_clause_list{$lbl2}); } elsif ($lbl1) { # Search for table.column in the subquery or function code my $tmp_str = $r; while ($tmp_str =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { if ($tmp_str =~ /\b([^\.\s]+\.[^\.\s]+)\.[^\.\s]+/ || $tmp_str =~ /\b([^\.\s]+)\.[^\.\s]+/) { $lbl2 = lc($1); # If the table/alias is not part of the from clause if (!exists $from_clause_list{$lbl2}) { $from_clause_list{$lbl2} = $lbl2; $from_order{$lbl2} = $fidx++; } $table_decl2 = $from_clause_list{$lbl2}; $table_decl2 .= " $lbl2" if ($lbl2 ne $from_clause_list{$lbl2}); } } if (!$lbl2 ) { push(@{$other_join_clause{$lbl1}}, "$l $o $r"); next; } } # When this is the first join parse add the left tablename # first then the outer join with the right table if (scalar keys %final_from_clause == 0) { $from_clause = $table_decl1; $table_decl1 =~ s/\s*\%ORA2PG_COMMENT\d+\%\s*//igs; push(@outer_clauses, (split(/\s/, $table_decl1))[1] || $table_decl1); $final_from_clause{"$lbl1;$lbl2"}{position} = $i; push(@{$final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{predicat}}, "$l $o $r"); } else { $final_from_clause{"$lbl1;$lbl2"}{position} = $i; push(@{$final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{predicat}}, "$l $o $r"); if (!exists $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{$type}) { $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{$type} = $table_decl1; } } if ($type eq 'left') { $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{position} = $i; } else { $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl1}{position} = $i; } } $str = $start_query . join(' ', @predicat) . ' ' . $end_query; # Remove part from the WHERE clause that will be moved into the FROM clause $str =~ s/\s*(AND\s+)?WHERE_CLAUSE\d+ / /igs; $str =~ s/WHERE\s+(AND|OR)\s+/WHERE /is; $str =~ s/WHERE[\s;]+$//i; $str =~ s/(\s+)WHERE\s+(ORDER|GROUP)\s+BY/$1$2 BY/is; $str =~ s/\s+WHERE(\s+)/\nWHERE$1/igs; my %associated_clause = (); foreach my $t (sort { $final_from_clause{$a}{position} <=> $final_from_clause{$b}{position} } keys %final_from_clause) { foreach my $j (sort { $final_from_clause{$t}{clause}{$a}{position} <=> $final_from_clause{$t}{clause}{$b}{position} } keys %{$final_from_clause{$t}{clause}}) { next if ($#{$final_from_clause{$t}{clause}{$j}{predicat}} < 0); if (exists $final_from_clause{$t}{clause}{$j}{$type} && $j !~ /\%SUBQUERY\d+\%/i && $from_clause !~ /\b\Q$final_from_clause{$t}{clause}{$j}{$type}\E\b/) { $from_clause .= ",$final_from_clause{$t}{clause}{$j}{$type}"; push(@outer_clauses, (split(/\s/, $final_from_clause{$t}{clause}{$j}{$type}))[1] || $final_from_clause{$t}{clause}{$j}{$type}); } my ($l,$r) = split(/;/, $t); my $tbl = $j; $tbl =~ s/\s*\%ORA2PG_COMMENT\d+\%\s*//isg; $from_clause .= "\n\U$type\E OUTER JOIN $tbl ON (" . join(' AND ', @{$final_from_clause{$t}{clause}{$j}{predicat}}) . ")"; push(@{$final_outer_clauses{$l}{join}}, "\U$type\E OUTER JOIN $tbl ON (" . join(' AND ', @{$final_from_clause{$t}{clause}{$j}{predicat}}, @{$other_join_clause{$r}}) . ")"); push(@{$final_outer_clauses{$l}{position}}, $final_from_clause{$t}{clause}{$j}{position}); push(@{$associated_clause{$l}}, $r); } } $from_clause = ''; my @clause_done = (); foreach my $c (sort { $from_order{$a} <=> $from_order{$b} } keys %from_order) { next if (!grep(/^\Q$c\E$/i, @outer_clauses)); my @output = (); for (my $j = 0; $j <= $#{$final_outer_clauses{$c}{join}}; $j++) { push(@output, $final_outer_clauses{$c}{join}[$j]); } find_associated_clauses($c, \@output, \%associated_clause, \%final_outer_clauses); if (!grep(/\QJOIN $from_clause_list{$c} $c \E/is, @clause_done)) { $from_clause .= "\n, $from_clause_list{$c}"; $from_clause .= " $c" if ($c ne $from_clause_list{$c}); } foreach (@output) { $from_clause .= "\n" . $_; } push(@clause_done, @output); delete $from_order{$c}; delete $final_outer_clauses{$c}; delete $associated_clause{$c}; } $from_clause =~ s/^\s*,\s*//s; # Append tables to from clause that was not involved into an outer join foreach my $a (sort keys %from_clause_list) { my $table_decl = "$from_clause_list{$a}"; $table_decl .= " $a" if ($a ne $from_clause_list{$a}); # Remove comment before searching it inside the from clause my $tmp_tbl = $table_decl; my $comment = ''; while ($tmp_tbl =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { $comment .= $1; } if ($from_clause !~ /(^|\s|,)\Q$tmp_tbl\E\b/is) { $from_clause = "$table_decl, " . $from_clause; } elsif ($comment) { $from_clause = "$comment " . $from_clause; } } $from_clause =~ s/\b(new|old)\b/\U$1\E/gs; $from_clause =~ s/,\s*$/ /s; $str =~ s/FROM FROM_CLAUSE/FROM $from_clause/s; } return $str; } sub find_associated_clauses { my ($c, $output, $associated_clause, $final_outer_clauses) = @_; foreach my $f (@{$associated_clause->{$c}}) { for (my $j = 0; $j <= $#{$final_outer_clauses->{$f}{join}}; $j++) { push(@$output, $final_outer_clauses->{$f}{join}[$j]); } delete $final_outer_clauses->{$f}; if (scalar keys %{ $final_outer_clauses }) { find_associated_clauses($f, $output, $associated_clause, $final_outer_clauses); } } delete $associated_clause->{$c}; } sub replace_connect_by { my ($class, $str) = @_; return $str if ($str !~ /\bCONNECT\s+BY\b/is); my $into_clause = ''; if ($str =~ s/\s+INTO\s+(.*?)(\s+FROM\s+)/$2/is) { $into_clause = " INTO $1"; } my $final_query = "WITH RECURSIVE cte AS (\n"; # Remove NOCYCLE, not supported at now $str =~ s/\s+NOCYCLE//is; # Remove SIBLINGS keywords and enable siblings rewrite my $siblings = 0; if ($str =~ s/\s+SIBLINGS//is) { $siblings = 1; } # Extract UNION part of the query to past it at end my $union = ''; if ($str =~ s/(CONNECT BY.*)(\s+UNION\s+.*)/$1/is) { $union = $2; } # Extract order by to past it to the query at end my $order_by = ''; if ($str =~ s/\s+ORDER BY(.*)//is) { $order_by = $1; } # Extract group by to past it to the query at end my $group_by = ''; if ($str =~ s/(\s+GROUP BY.*)//is) { $group_by = $1; } # Extract the starting node or level of the tree my $where_clause = ''; my $start_with = ''; if ($str =~ s/WHERE\s+(.*?)\s+START\s+WITH\s*(.*?)\s+CONNECT BY\s*//is) { $where_clause = " WHERE $1"; $start_with = $2; } elsif ($str =~ s/WHERE\s+(.*?)\s+CONNECT BY\s+(.*?)\s+START\s+WITH\s*(.*)/$2/is) { $where_clause = " WHERE $1"; $start_with = $3; } elsif ($str =~ s/START\s+WITH\s*(.*?)\s+CONNECT BY\s*//is) { $start_with = $1; } elsif ($str =~ s/\s+CONNECT BY\s+(.*?)\s+START\s+WITH\s*(.*)/ $1 /is) { $start_with = $2; } else { $str =~ s/CONNECT BY\s*//is; } # remove alias from where clause $where_clause =~ s/\b[^\.]\.([^\s]+)\b/$1/gs; # Extract the CONNECT BY clause in the hierarchical query my $prior_str = ''; my @prior_clause = ''; if ($str =~ s/([^\s]+\s*=\s*PRIOR\s+.*)//is) { $prior_str = $1; } elsif ($str =~ s/(\s*PRIOR\s+.*)//is) { $prior_str = $1; } else { # look inside subqueries if we have a prior clause my @ids = $str =~ /\%SUBQUERY(\d+)\%/g; my $sub_prior_str = ''; foreach my $i (@ids) { if ($class->{sub_parts}{$i} =~ s/([^\s]+\s*=\s*PRIOR\s+.*)//is) { $sub_prior_str = $1; $str =~ s/\%SUBQUERY$i\%//; } elsif ($class->{sub_parts}{$i} =~ s/(\s*PRIOR\s+.*)//is) { $sub_prior_str = $1; $str =~ s/\%SUBQUERY$i\%//; } $sub_prior_str =~ s/^\(//; $sub_prior_str =~ s/\)$//; ($prior_str ne '' || $sub_prior_str eq '') ? $prior_str .= ' ' . $sub_prior_str : $prior_str = $sub_prior_str; } } if ($prior_str) { # Try to extract the prior clauses my @tmp_prior = split(/\s*AND\s*/, $prior_str); $tmp_prior[-1] =~ s/\s*;\s*//s; my @tmp_prior2 = (); foreach my $p (@tmp_prior) { if ($p =~ /\bPRIOR\b/is) { push(@prior_clause, split(/\s*=\s*/i, $p)); } else { $where_clause .= " AND $p"; } } if ($siblings) { if ($prior_clause[-1] !~ /PRIOR/i) { $siblings = $prior_clause[-1]; } else { $siblings = $prior_clause[-2]; } $siblings =~ s/\s+//g; } shift(@prior_clause) if ($prior_clause[0] eq ''); my @rebuild_prior = (); # Place PRIOR in the left part if necessary for (my $i = 0; $i < $#prior_clause; $i+=2) { if ($prior_clause[$i+1] =~ /PRIOR\s+/i) { my $tmp = $prior_clause[$i]; $prior_clause[$i] = $prior_clause[$i+1]; $prior_clause[$i+1] = $tmp; } push(@rebuild_prior, "$prior_clause[$i] = $prior_clause[$i+1]"); } @prior_clause = @rebuild_prior; # Remove table aliases from prior clause map { s/\s*PRIOR\s*//s; s/[^\s\.=<>!]+\.//s; } @prior_clause; } my $bkup_query = $str; # Construct the initialization query $str =~ s/(SELECT\s+)(.*?)(\s+FROM)/$1COLUMN_ALIAS$3/is; my @columns = split(/\s*,\s*/, $2); # When the pseudo column LEVEL is used in the where clause # and not used in columns list, add the pseudo column if ($where_clause =~ /\bLEVEL\b/is && !grep(/\bLEVEL\b/i, @columns)) { push(@columns, 'level'); } my @tabalias = (); my %connect_by_path = (); for (my $i = 0; $i <= $#columns; $i++) { my $found = 0; while ($columns[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { # Get out of here next run when a call to SYS_CONNECT_BY_PATH is found # This will prevent opening too much subquery in the function parameters last if ($found); $found = 1 if ($columns[$i]=~ /SYS_CONNECT_BY_PATH/is); }; # Replace LEVEL call by a counter, there is no direct equivalent in PostgreSQL if (lc($columns[$i]) eq 'level') { $columns[$i] = "1 as level"; } elsif ($columns[$i] =~ /\bLEVEL\b/is) { $columns[$i] =~ s/\bLEVEL\b/1/is; } # Replace call to SYS_CONNECT_BY_PATH by the right concatenation string if ($columns[$i] =~ s/SYS_CONNECT_BY_PATH\s*[\(]*\s*([^,]+),\s*([^\)]+)\s*\)/$1/is) { my $col = $1; $connect_by_path{$col}{sep} = $2; # get the column alias if ($columns[$i] =~ /\s+([^\s]+)\s*$/s) { $connect_by_path{$col}{alias} = $1; } } if ($columns[$i] =~ /([^\.]+)\./s) { push(@tabalias, $1) if (!grep(/^\Q$1\E$/i, @tabalias)); } extract_subpart($class, \$columns[$i]); # Append parenthesis on new subqueries values foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) { next if ($class->{sub_parts}{$z} =~ /^\(/ || $class->{sub_parts}{$z} =~ /^TABLE[\(\%]/i); # If subpart is not empty after transformation if ($class->{sub_parts}{$z} =~ /\S/is) { # add open and closed parenthesis $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) { # otherwise do not report the empty parenthesis when this is not a function $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } } } # Extraction of the table aliases in the FROM clause my $cols = join(',', @columns); $str =~ s/COLUMN_ALIAS/$cols/s; if ($str =~ s/(\s+FROM\s+)(.*)/$1FROM_CLAUSE/is) { my $from_clause = $2; $str =~ s/FROM_CLAUSE/$from_clause/; } # Now append the UNION ALL query that will be called recursively if ($str =~ s/^(\s*BEGIN\s+(?:.*)?(?:\s+))(SELECT\s+)/$2/is) { $final_query = "$1$final_query"; $final_query .= $str; $bkup_query =~ s/^(\s*BEGIN\s+(?:.*)?(?:\s+))(SELECT\s+)/$2/is; } else { $final_query .= $str; } $final_query .= ' WHERE ' . $start_with . "\n" if ($start_with); #$where_clause =~ s/^\s*WHERE\s+/ AND /is; #$final_query .= $where_clause . "\n"; $final_query .= " UNION ALL\n"; if ($siblings && !$order_by) { $final_query =~ s/(\s+FROM\s+)/,ARRAY[ row_number() OVER (ORDER BY $siblings) ] as hierarchy$1/is; } elsif ($siblings) { $final_query =~ s/(\s+FROM\s+)/,ARRAY[ row_number() OVER (ORDER BY $order_by) ] as hierarchy$1/is; } $bkup_query =~ s/(SELECT\s+)(.*?)(\s+FROM)/$1COLUMN_ALIAS$3/is; @columns = split(/\s*,\s*/, $2); # When the pseudo column LEVEL is used in the where clause # and not used in columns list, add the pseudo column if ($where_clause =~ /\bLEVEL\b/is && !grep(/\bLEVEL\b/i, @columns)) { push(@columns, 'level'); } for (my $i = 0; $i <= $#columns; $i++) { my $found = 0; while ($columns[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { # Get out of here when a call to SYS_CONNECT_BY_PATH is found # This will prevent opening subquery in the function parameters last if ($found); $found = 1 if ($columns[$i]=~ /SYS_CONNECT_BY_PATH/is); }; if ($columns[$i] =~ s/SYS_CONNECT_BY_PATH\s*[\(]*\s*([^,]+),\s*([^\)]+)\s*\)/$1/is) { $columns[$i] = "c.$connect_by_path{$1}{alias} || $connect_by_path{$1}{sep} || " . $columns[$i]; } if ($columns[$i] !~ s/\b[^\.]+\.LEVEL\b/(c.level+1)/igs) { $columns[$i] =~ s/\bLEVEL\b/(c.level+1)/igs; } extract_subpart($class, \$columns[$i]); # Append parenthesis on new subqueries values foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) { next if ($class->{sub_parts}{$z} =~ /^\(/ || $class->{sub_parts}{$z} =~ /^TABLE[\(\%]/i); # If subpart is not empty after transformation if ($class->{sub_parts}{$z} =~ /\S/is) { # add open and closed parenthesis $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) { # otherwise do not report the empty parenthesis when this is not a function $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; } } } $cols = join(',', @columns); $bkup_query =~ s/COLUMN_ALIAS/$cols/s; my $prior_alias = ''; if ($bkup_query =~ s/(\s+FROM\s+)(.*)/$1FROM_CLAUSE/is) { my $from_clause = $2; if ($from_clause =~ /\b[^\s]+\s+(?:AS\s+)?([^\s]+)\b/) { my $a = $1; $prior_alias = "$a." if (!grep(/\b$a\.[^\s]+$/, @prior_clause)); } $bkup_query =~ s/FROM_CLAUSE/$from_clause/; } # Remove last subquery alias in the from clause to put our own $bkup_query =~ s/(\%SUBQUERY\d+\%)\s+[^\s]+\s*$/$1/is; if ($siblings && $order_by) { $bkup_query =~ s/(\s+FROM\s+)/, array_append(c.hierarchy, row_number() OVER (ORDER BY $order_by)) as hierarchy$1/is; } elsif ($siblings) { $bkup_query =~ s/(\s+FROM\s+)/, array_append(c.hierarchy, row_number() OVER (ORDER BY $siblings)) as hierarchy$1/is; } $final_query .= $bkup_query; map { s/^\s*(.*?)(=\s*)(.*)/c\.$1$2$prior_alias$3/s; } @prior_clause; map { s/\s+$//s; s/^\s+//s; } @prior_clause; $final_query .= " JOIN cte c ON (" . join(' AND ', @prior_clause) . ")\n"; if ($siblings) { $order_by = " ORDER BY hierarchy"; } elsif ($order_by) { $order_by =~ s/^, //s; $order_by = " ORDER BY $order_by"; } $final_query .= "\n) SELECT *$into_clause FROM cte$where_clause$union$group_by$order_by"; return $final_query; } sub replace_without_function { my ($class, $str) = @_; # Code disabled because it break other complex GROUP BY clauses # Keeping it just in case some light help me to solve this problem # Reported in issue #496 # Remove text constant in GROUP BY clause, this is not allowed # GROUP BY ?TEXTVALUE10?, %%REPLACEFCT1%%, DDI.LEGAL_ENTITY_ID #if ($str =~ s/(\s+GROUP\s+BY\s+)(.*?)((?:(?=\bUNION\b|\bORDER\s+BY\b|\bLIMIT\b|\bINTO\s+|\bFOR\s+UPDATE\b|\bPROCEDURE\b).)+|$)/$1\%GROUPBY\% $3/is) { # my $tmp = $2; # $tmp =~ s/\?TEXTVALUE\d+\?[,]*\s*//gs; # $tmp =~ s/(\s*,\s*),\s*/$1/gs; # $tmp =~ s/\s*,\s*$//s; # $str =~ s/\%GROUPBY\%/$tmp/s; #} return $str; } =head2 mssql_to_plpgsql This function turn a MSSQL function code into a PLPGSQL code =cut sub mssql_to_plpgsql { my ($class, $str) = @_; # Replace getdate() with CURRENT_TIMESTAMP $str =~ s/\bgetdate\s*\(\s*\)/date_trunc('millisecond', CURRENT_TIMESTAMP::timestamp)/ig; $str =~ s/\bgetutcdate\s*\(\s*\)/date_trunc('millisecond', now() AT TIME ZONE 'UTC')/ig; # Replace user_name() with CURRENT_USER $str =~ s/\buser_name\s*\(\s*\)/CURRENT_USER/gi; # Remove call to with(nolock) from queries $str =~ s/\bwith\s*\(\s*nolock\s*\)//ig; $str =~ s/\bwith\s*schemabinding//ig; # Replace call to SYS_GUID() function $str =~ s/\bnewid\s*\(\s*\)/$class->{uuid_function}()/ig; # Remove COUNT setting $str =~ s/SET NOCOUNT (ON|OFF)[;\s]*//ig; # Remove quoted definer order $str =~ s/SET QUOTED_IDENTIFIER (ON|OFF)//igs; # Replace BREAK by EXIT $str =~ s/\bBREAK\s*[;]*$/EXIT;/ig; # Rewrite call to sequences while ($str =~ /NEXT VALUE FOR ([^\s]+)/i) { my $seqname = $1; $seqname =~ s/[\[\]\)]+//g; $str =~ s/[\(]*NEXT VALUE FOR ([^\s]+)/nextval('$seqname')/i; } #### # Replace some function with their PostgreSQL syntax #### $str =~ s/\bDATALENGTH\s*\(/LENGTH(/gi; $str =~ s/\bLEN\s*\(([^\)]+)\)/LENGTH(RTRIM($1))/gi; $str =~ s/ISNULL\s*\(/COALESCE(/gi; $str =~ s/SPACE\s*\(/REPEAT(' ', /gi; $str =~ s/REPLICATE\s*\(/REPEAT(/gi; $str =~ s/CHARINDEX\s*\(\s*(.*?)\s*,\s*(.*?)\s*,\s*(\d+)\)/position('$1' in substring($2 from $3))/gi; $str =~ s/CHARINDEX\s*\(\s*(.*?)\s*,\s*(.*?)\s*\)/position('$1' in $2)/gi; $str =~ s/DATEPART\s*\(\s*(.*?)\s*,\s*(.*?)\s*\)/date_part('$1', $2)/gi; $str =~ s/DATEADD\s*\(\s*(.*?)\s*\,\s*(.*?)\s*,\s*(.*?)\s*\)/$3 + INTERVAL '$2 $1'/gi; $str =~ s/CONVERT\s*\(\s*(.*?)\s*,\s*(.*?)\s*,\s*(\d+)\)/TO_CHAR($2, '$MSSQL_STYLE{$3}')::$1/gi; $str =~ s/CONVERT\s*\(\s*NVARCHAR\s*(.*?)\s*\(\s*(.*?)\s*\s*\)\,\s*(.*?)\s*\)/CAST($3 AS varchar($2))/gi; $str =~ s/CONVERT\s*\(\s*(.*?)\s*\(\s*(.*?)\s*\s*\),\s*(.*?)\s*\)/CAST($3 AS $1($2))/gi; $str =~ s/CONVERT\s*\(\s*(.*?)\s*\,\s*(.*?)\s*\)/CAST($2 AS $1)/gi; $str =~ s/\bRAND\s*\(/random(/gi; $str =~ s/\bYEAR\s*\(/date_part('year', /gi; $str =~ s/\bMONTH\s*\(/date_part('month', /gi; $str =~ s/\bDAY\s*\(/date_part('day', /gi; $str =~ s/\bSYSDATETIMEOFFSET\s*\(/now(/gi; $str =~ s/\bSYSDATETIME\s*\(\s*\)/now()::timestamp/gi; $str =~ s/\bSYSUTCDATETIME\s*\(\s*\)/now() at time zone 'UTC'/gi; $str =~ s/\bDATENAME\s*\(\s*(?:weekday|dw|w)\s*,\s*([^\)]+)\s*\)/to_char($1, 'day')/gi; $str =~ s/\bDATENAME\s*\(\s*(?:dayofyear|dy|y)\s*,\s*([^\)]+)\s*\)/date_part('doy', $1)/gi; $str =~ s/\bDATENAME\s*\(\s*(?:month|mm|m)\s*,\s*([^\)]+)\s*\)/to_char($1, 'month')/gi; $str =~ s/\bDATENAME\s*\(\s*(?:year|yy|y)\s*,/date_part('year',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:quarter|qq|q)\s*,/date_part('quarter',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:day|dd|d)\s*,/date_part('day',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:week|ww|wk)\s*,\s*([^\)]+)\s*\)/date_part('week', $1)+1/gi; $str =~ s/\bDATENAME\s*\(\s*hour\s*,/date_part('hour',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:minute|mi|n)\s*,/date_part('minute',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:second|ss|s)\s*,/date_part('second',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:millisecond|ms)\s*,/date_part('millisecond',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:microsecond|mcs)\s*,/date_part('microsecond',/gi; $str =~ s/\bDATENAME\s*\(\s*(?:nanosecond|ns)\s*,/date_part('nanosecond',/gi; # will fail, should be microsecond $str =~ s/\bDATENAME\s*\(\s*(?:ISO_WEEK|ISOWK|ISOWW)\s*,\s*([^\)]+)\s*\)/date_part('week', $1)/gi; $str =~ s/\bDATENAME\s*\(\s*(?:TZoffset|tz)\s*,\s*([^\)]+)\s*\)/to_char($1, 'TZH:TZM')/gi; # Rewrite expression like SET p_MatchExpression = '%'+p_MatchExpression+'%' $str =~ s/\bSET\s+([^\s=;:]+)\s*=/$1 :=/igs; # Fix IF ... BEGIN into IF ... THEN on single line $str =~ s/(\s+IF[\s\(]+(?:.*?))\s+BEGIN\b/$1 THEN/igs; # Fix WHILE ... BEGIN into IF ... THEN $str =~ s/(\s+WHILE[\s\(]+(?:.*?))\s+BEGIN\b/$1 LOOP/igs; # Fix ELSE IF into ELSIF $str =~ s/\bELSE\s+IF\b/ELSIF/igs; # Fix temporary table creation. We keep the # so that they can be identified in the code $str =~ s/CREATE\s+TABLE\s+#/CREATE TEMPORARY TABLE #/igs; # Replace identity function $str =~ s/int identity\(\s*([\d+])\s*,\s*([\d+])\s*\)/int GENERATED BY DEFAULT AS IDENTITY (START WITH $1 INCREMENT BY $2)/igs; return $str; } 1; __END__ =head1 AUTHOR Gilles Darold =head1 COPYRIGHT Copyright (c) 2000-2025 Gilles Darold - All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. =head1 BUGS This perl module is in the same state as my knowledge regarding database, it can move and not be compatible with older version so I will do my best to give you official support for Ora2Pg. Your volontee to help construct it and your contribution are welcome. =head1 SEE ALSO L =cut ora2pg-25.0/packaging/000077500000000000000000000000001500113072400145475ustar00rootroot00000000000000ora2pg-25.0/packaging/README000066400000000000000000000023631500113072400154330ustar00rootroot00000000000000RPM/ Holds ora2pg.spec need to build an RPM package for RH/Fedora. It may also be usable for other RPM based distribution. Copy the ora2pg source tarball here: ~/rpmbuild/SOURCES/ or /usr/src/redhat/SOURCES/ Then create the RPM binary package as follow: rpmbuild -bb ora2pg.spec The binary package may be found here: ~/rpmbuild/RPMS/noarch/ora2pg-25.0-1.noarch.rpm or /usr/src/redhat/RPMS/i386/ora2pg-25.0-1.noarch.rpm To install run: rpm -i ~/rpmbuild/RPMS/noarch/ora2pg-25.0-1.noarch.rpm slackbuild/ Holds all files necessary to build a Slackware package. Copy the source tarball into the slackbuild directory and run sh Ora2Pg.SlackBuild then take a look at /tmp/build/ to find the Slackware package. To install run the following command: installpkg /tmp/build/ora2pg-25.0-i386-1gda.tgz or installpkg /tmp/build/ora2pg-25.0-x86_64-1gda.tgz following the architecture. debian/ Holds all files to build debian package. First you need to execute script 'sh create-deb-tree.sh' in the debian directory to create the package tree. After that just run the following command to generate the debian package: dpkg -b ora2pg ora2pg.deb To install the package, run: dpkg -i ora2pg.deb Feel free to send me other. ora2pg-25.0/packaging/RPM/000077500000000000000000000000001500113072400152055ustar00rootroot00000000000000ora2pg-25.0/packaging/RPM/ora2pg.spec000066400000000000000000000056251500113072400172630ustar00rootroot00000000000000Summary: Oracle to PostgreSQL database schema converter Name: ora2pg Version: 18.0 Release: 1%{?dist} Group: Applications/Databases License: GPLv3+ URL: http://ora2pg.darold.net/ Source0: https://github.com/darold/%{name}/archive/v%{version}.tar.gz BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: perl Requires: perl(DBD::Oracle) Requires: perl-DBD-MySQL perl(DBI) perl(IO::Compress::Base) %description This package contains a Perl module and a companion script to convert an Oracle database schema to PostgreSQL and to migrate the data from an Oracle database to a PostgreSQL database. %prep %setup -q %build # Make Perl and Ora2Pg distrib files %{__perl} Makefile.PL \ INSTALLDIRS=vendor \ QUIET=1 \ CONFDIR=%{_sysconfdir} \ DOCDIR=%{_docdir}/%{name}-%{version} \ DESTDIR=%{buildroot} %{__make} %install %{__rm} -rf %{buildroot} %{__make} install DESTDIR=%{buildroot} # Remove unpackaged files. %{__rm} -f `find %{buildroot}/%{_libdir}/perl*/ -name .packlist -type f` %{__rm} -f `find %{buildroot}/%{_libdir}/perl*/ -name perllocal.pod -type f` %clean %{__rm} -rf %{buildroot} %files %defattr(-, root, root, 0755) %attr(0755,root,root) %{_bindir}/%{name} %attr(0755,root,root) %{_bindir}/%{name}_scanner %attr(0644,root,root) %{_mandir}/man3/%{name}.3.gz %config(noreplace) %{_sysconfdir}/%{name}.conf.dist #%config(noreplace) %{_sysconfdir}/%{name}/%{name}.conf.dist %{perl_vendorlib}/Ora2Pg/MySQL.pm %{perl_vendorlib}/Ora2Pg/PLSQL.pm %{perl_vendorlib}/Ora2Pg/GEOM.pm %{perl_vendorlib}/Ora2Pg.pm %{_docdir}/%{name}-%{version}/* %changelog * Tue Jan 31 2017 Devrim Gündüz 18.0-1 - Update to 18.0 * Mon Nov 21 2016 Devrim Gündüz 17.6-1 - Update to 17.6 * Fri Oct 21 2016 Devrim Gündüz 17.5-1 - Update to 17.5 * Mon Apr 18 2016 Devrim Gündüz 17.3-1 - Update to 17.3 * Fri Mar 25 2016 Devrim Gündüz 17.2-1 - Update to 17.2 * Wed Mar 9 2016 Devrim Gündüz 17.1-1 - Update to 17.1 * Thu Jan 21 2016 Devrim Gündüz 16.2-1 - Update to 16.2 * Wed Dec 30 2015 Devrim GUNDUZ 16.1-1 - Update to 16.1 * Fri Feb 6 2015 Devrim GUNDUZ 15.1-1 - Update to 15.1, per changes described at: http://www.postgresql.org/message-id/54D49C0B.2000006@dalibo.com * Wed Oct 23 2013 Devrim GUNDUZ 12.0-1 - Update to 12.0, per changes described at: http://www.postgresql.org/message-id/52664854.30200@dalibo.com * Thu Sep 12 2013 Devrim GUNDUZ 11.4-1 - Update to 11.4 * Thu Sep 13 2012 Devrim GUNDUZ 9.2-1 - Update to 9.2 - Update URL, License, Group tags - Fix spec per rpmlint warnings - Apply some changes from upstream spec * Fri Mar 20 2009 Devrim GUNDUZ 5.0-1 - Initial release, based on Peter's spec file. ora2pg-25.0/packaging/debian/000077500000000000000000000000001500113072400157715ustar00rootroot00000000000000ora2pg-25.0/packaging/debian/create-deb-tree.sh000066400000000000000000000007751500113072400212660ustar00rootroot00000000000000#!/bin/sh # # Script used to create the Debian package tree. This script must be # executed in his directory # LPWD=`pwd` DEST=packaging/debian/ora2pg cd ../../ perl Makefile.PL \ INSTALLDIRS=vendor \ QUIET=1 \ CONFDIR=/etc/ora2pg \ DOCDIR=/usr/share/doc/ora2pg \ DESTDIR=$DEST || exit 1 make && make install DESTDIR=$DEST echo "Compressing man pages" find $DEST/usr/share/man/ -type f -name "*.?" -exec gzip -9 {} \; find $DEST/usr/share/man/ -type f -name "*.?pm" -exec gzip -9 {} \; cd $LPWD ora2pg-25.0/packaging/debian/ora2pg/000077500000000000000000000000001500113072400171635ustar00rootroot00000000000000ora2pg-25.0/packaging/debian/ora2pg/DEBIAN/000077500000000000000000000000001500113072400201055ustar00rootroot00000000000000ora2pg-25.0/packaging/debian/ora2pg/DEBIAN/control000066400000000000000000000010171500113072400215070ustar00rootroot00000000000000Package: ora2pg Version: 25.0 Priority: optional Architecture: all Essential: no Depends: libcompress-zlib-perl, libdbd-oracle-perl, libdbi-perl, libdbd-mysql-perl Pre-Depends: perl Recommends: postgresql-client Installed-Size: 1024 Maintainer: Gilles Darold Provide: ora2pg Description: Ora2Pg (Oracle to PostgreSQL database schema converter) This package contains all Perl modules and scripts to convert an Oracle or MySQL database schema, data and stored procedures to a PostgreSQL database. ora2pg-25.0/packaging/debian/ora2pg/DEBIAN/copyright000066400000000000000000000025731500113072400220470ustar00rootroot00000000000000This work was packaged for Debian by: Peter Eisentraut on 2007-10-24 Julin Moreno Patio on Sun, 21 Mar 2010 07:18:35 -0500 It was downloaded from: http://ora2pg.darold.net/ Upstream Author(s): Gilles Darold Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved License: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . On Debian systems, the complete text of the GNU General Public License version 3 can be found in "/usr/share/common-licenses/GPL-3". The Debian packaging is: Copyright (C) 2007 Peter Eisentraut Copyright (C) 2009 Julin Moreno Patio and is licensed under the GPL version 2, see `/usr/share/common-licenses/GPL-2'. ora2pg-25.0/packaging/slackbuild/000077500000000000000000000000001500113072400166645ustar00rootroot00000000000000ora2pg-25.0/packaging/slackbuild/Ora2Pg.SlackBuild000066400000000000000000000102321500113072400217130ustar00rootroot00000000000000#!/bin/sh # Written by Gilles Darold < gilles at darold dot net > # Licence: GPL v3 # # Build script for Slackware - Ora2Pg SlackBuild # # Latest Software sourcecode is available at: # http://ora2pg.darold.net/ # # Depends: d/perl # Suggests: ## Fill these variables to your needs ## NAMESRC=${NAMESRC:-ora2pg} VERSION=${VERSION:-25.0} EXT=${EXT:-tar.bz2} NAMEPKG=${NAMEPKG:-ora2pg} PKGEXT=${PKGEXT:-tgz/txz} BUILD=${BUILD:-1} TAG=${TAG:-_gda} PNAME=${PNAME:-ora2pg} TMP=${TMP:-/tmp} OUT=${OUT:-$TMP/build} ARCH=${ARCH:-i486} TARGET=${TARGET:-i486} WGET=${WGET:-http://downloads.sourceforge.net/ora2pg/$NAMESRC-$VERSION.$EXT} DOC="Change* INSTALL README" SUM="" ######################################## set -e umask 022 if [ ! -r $NAMESRC-$VERSION.$EXT ]; then wget -vc $WGET -O $NAMESRC-$VERSION.$EXT.part mv $NAMESRC-$VERSION.$EXT.part $NAMESRC-$VERSION.$EXT fi # if checksum is include in the script : generate and check if [ -n "$SUM" ]; then echo "$SUM $NAMESRC-$VERSION.$EXT" > $NAMESRC-$VERSION.$EXT.sha1 sha1sum -c $NAMESRC-$VERSION.$EXT.sha1 elif [ -f $NAMESRC-$VERSION.$EXT.sha1 ]; then sha1sum -c $NAMESRC-$VERSION.$EXT.sha1 fi # or just check if the .sha1 is another file CWD=$(pwd) PKG=$TMP/build/$NAMEPKG NAME=$(tar ft $NAMESRC-$VERSION.$EXT | head -n 1 | awk -F/ '{ print $1 }') case $ARCH in i386)SLKCFLAGS="-O2 -march=i386 -mtune=i686";LIBDIRSUFFIX="";; i486)SLKCFLAGS="-O2 -march=i486 -mtune=i686";LIBDIRSUFFIX="";; i586)SLKCFLAGS="-O2 -march=i586 -mtune=i686";LIBDIRSUFFIX="";; i686)SLKCFLAGS="-O2 -march=i686 -mtune=i686";LIBDIRSUFFIX="";; s390)SLKCFLAGS="-O2";LIBDIRSUFFIX="";; x86_64)SLKCFLAGS="-O2 -fPIC";LIBDIRSUFFIX="64" esac if [ "$(id -u)" = "0" ]; then echo "You shouldn't run this SlackBuild as ROOT !" exit 1 fi if [ ! -d $TMP ]; then echo "$TMP doesn't exist or is not a directory !" exit 1 fi # Build the software cd $TMP echo "Building $NAMESRC-$VERSION.$EXT..." tar xf $CWD/$NAMESRC-$VERSION.$EXT cd $NAME perl Makefile.PL \ INSTALLDIRS=vendor \ QUIET=1 \ CONFDIR=/etc/$PNAME \ DOCDIR=/usr/share/doc/$PNAME \ DESTDIR=$PKG || exit 1 make make install DESTDIR=$PKG # Please note that some software use INSTALL_ROOT=$PKG or prefix=$PKG/usr or install_root=$PKG ... # Install a slack-desc mkdir -p $PKG/install cat $CWD/slack-desc > $PKG/install/slack-desc # Install a doinst.sh, if it exists if [ -r $CWD/doinst.sh ]; then cat $CWD/doinst.sh > $PKG/install/doinst.sh fi mkdir -p $PKG/usr/doc/$PNAME-$VERSION cp -a $DOC $PKG/usr/doc/$PNAME-$VERSION # Compress the man pages if [ -d $PKG/usr/man ]; then find $PKG/usr/man -type f -name "*.?" -exec gzip -9 {} \; for manpage in $(find $PKG/usr/man -type l) ; do ln -s $(readlink $manpage).gz $manpage.gz rm -f $manpage done fi if [ -d $PKG/usr/share/man ]; then find $PKG/usr/share/man -type f -name "*.?" -exec gzip -9 {} \; for manpage in $(find $PKG/usr/share/man -type l) ; do ln -s $(readlink $manpage).gz $manpage.gz rm -f $manpage done fi # Compress the info pages if [ -d $PKG/usr/info ]; then rm -f $PKG/usr/info/dir gzip -9 $PKG/usr/info/*.info* fi # Remove 'special' files find $PKG -name perllocal.pod \ -o -name ".packlist" \ -o -name "*.bs" \ | xargs rm -f # Remove empty directory rmdir --parents $PKG/usr/lib/perl5/5.*/i486-linux-thread-multi 2>/dev/null || true rmdir --parents $PKG/usr/lib/perl5/vendor_perl/5.*/i486-linux-thread-multi/auto/Ora2Pg 2>/dev/null || true # Strip binaries, libraries and archives find $PKG -type f | xargs file | grep "LSB executable" | cut -f 1 -d : | xargs \ strip --strip-unneeded 2> /dev/null || echo "No binaries to strip" find $PKG -type f | xargs file | grep "shared object" | cut -f 1 -d : | xargs \ strip --strip-unneeded 2> /dev/null || echo "No shared objects to strip" find $PKG -type f | xargs file | grep "current ar archive" | cut -f 1 -d : | \ xargs strip -g 2> /dev/null || echo "No archives to strip" # Build the package cd $PKG mkdir -p $OUT PACKAGING=" chown root:root . -R /sbin/makepkg -l y -c n $OUT/$NAMEPKG-$VERSION-$ARCH-$BUILD$TAG.tgz rm -rf $PKG rm -rf $TMP/$NAME " if [ "$(which fakeroot 2> /dev/null)" ]; then echo "$PACKAGING" | fakeroot else su -c "$PACKAGING" fi ora2pg-25.0/packaging/slackbuild/Ora2Pg.info000066400000000000000000000004011500113072400206260ustar00rootroot00000000000000PRGNAM="Ora2Pg" VERSION="25.0" HOMEPAGE="http://ora2pg.darold.net/" DOWNLOAD="http://downloads.sourceforge.net/ora2pg/ora2pg-25.0.tar.gz" MD5SUM="" DOWNLOAD_x86_64="UNTESTED" MD5SUM_x86_64="" MAINTAINER="Gilles Darold" EMAIL="gilles@darold.net" APPROVED="" ora2pg-25.0/packaging/slackbuild/README000066400000000000000000000021561500113072400175500ustar00rootroot00000000000000Ora2Pg is a tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scan it automaticaly and extracts its structure or data, it then generates SQL scripts that you can load into your PostgreSQL database. Ora2Pg can be used from reverse engineering Oracle database to huge enterprise database migration or simply to replicate some Oracle data into a PostgreSQL database. It is really easy to used and doesn't need any Oracle database knowledge than providing the parameters needed to connect to the Oracle database. You need a modern Perl distribution (perl 5.6 or more), the DBI and DBD::Oracle Perl modules to be installed. These are used to connect to the Oracle database. To install DBD::Oracle and have it working you need to have the Oracle client libraries installed and the ORACLE_HOME environment variable must be defined. Note that the Oracle and the PostgreSQL databases doesn't need to be on the host running Ora2Pg but this host must have at least the Oracle client libraries installed and the PostgreSQL client if you want to use psql or DBD::Pg to import data. ora2pg-25.0/packaging/slackbuild/doinst.sh000066400000000000000000000000311500113072400205120ustar00rootroot00000000000000#! /bin/sh cat README ora2pg-25.0/packaging/slackbuild/slack-desc000066400000000000000000000010761500113072400206240ustar00rootroot00000000000000 |-----handy-ruler------------------------------------------------------| Ora2Pg: Ora2Pg (Oracle to PostgreSQL database schema converter) Ora2Pg: Ora2Pg: This package contains Perl modules and a companion script to convert an Ora2Pg: Oracle database schema to PostgreSQL and to migrate the data from an Ora2Pg: Oracle database to a PostgreSQL database. Ora2Pg: Ora2Pg: Ora2Pg can be used from reverse engineering Oracle database to huge Ora2Pg: enterprise database migration or simply to replicate some Oracle data Ora2Pg: into a PostgreSQL database. Ora2Pg: Ora2Pg: ora2pg-25.0/scripts/000077500000000000000000000000001500113072400143125ustar00rootroot00000000000000ora2pg-25.0/scripts/ora2pg000077500000000000000000001374721500113072400154500ustar00rootroot00000000000000#!/usr/bin/env perl #------------------------------------------------------------------------------ # Project : Oracle to Postgresql converter # Name : ora2pg # Author : Gilles Darold, gilles _AT_ darold _DOT_ net # Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved - # Function : Script used to convert Oracle Database to PostgreSQL # Usage : ora2pg configuration_file #------------------------------------------------------------------------------ # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see < http://www.gnu.org/licenses/ >. # #------------------------------------------------------------------------------ use strict qw/vars/; use Ora2Pg; use Getopt::Long qw(:config no_ignore_case bundling); use File::Spec; use POSIX qw(locale_h sys_wait_h _exit); setlocale(LC_NUMERIC, ''); setlocale(LC_ALL, 'C'); my $VERSION = $Ora2Pg::VERSION; $| = 1; my $CONFIG_FILE = "/etc/ora2pg/ora2pg.conf"; my $FILE_CONF = ''; my $DEBUG = 0; my $QUIET = 0; my $HELP = 0; my $LOGFILE = ''; my $EXPORT_TYPE = ''; my $OUTFILE = ''; my $OUTDIR = ''; my $SHOW_VER = 0; my $PLSQL = ''; my $DSN = ''; my $DBUSER = ''; my $DBPWD = ''; my $SCHEMA = ''; my $TABLEONLY = ''; my $FORCEOWNER = ''; my $ORA_ENCODING = ''; my $PG_ENCODING = ''; my $INPUT_FILE = ''; my $EXCLUDE = ''; my $ALLOW = ''; my $VIEW_AS_TABLE = ''; my $MVIEW_AS_TABLE = ''; my $ESTIMATE_COST; my $COST_UNIT_VALUE = 5; my $DUMP_AS_HTML; my $DUMP_AS_CSV; my $DUMP_AS_JSON; my $DUMP_AS_FILE_PREFIX; my $DUMP_AS_SHEET; my $THREAD_COUNT; my $ORACLE_COPIES; my $PARALLEL_TABLES; my $DATA_LIMIT; my $CREATE_PROJECT = ''; my $PROJECT_BASE = '.'; my $PRINT_HEADER = ''; my $HUMAN_DAY_LIMIT; my $IS_MYSQL = 0; my $IS_MSSQL = 0; my $AUDIT_USER = ''; my $PG_DSN = ''; my $PG_USER = ''; my $PG_PWD = ''; my $COUNT_ROWS = 0; my $DATA_TYPE = ''; my $GRANT_OBJECT = ''; my $PG_SCHEMA = ''; my $NO_HEADER = 0; my $ORACLE_SPEED = 0; my $ORA2PG_SPEED = 0; my $RELATIVE_PATH = 0; my $BLOB_TO_LO = 0; my $IMPORT_LO = 0; my @WHERE_CLAUSE = (); my @DELETE_CLAUSE = (); my $START_SCN = ''; my $CDC_READY = ''; my $CDC_FILE = 'TABLES_SCN.log'; my $DROP_IF_EXISTS = 0; my $ORACLE_FDW_PREFETCH; my $NO_START_SCN = 0; my $NO_CLEAN_COMMENT = 0; my @SCHEMA_ARRAY = qw( SEQUENCE SEQUENCE_VALUES TABLE PACKAGE VIEW GRANT TRIGGER FUNCTION PROCEDURE TABLESPACE PARTITION TYPE MVIEW DBLINK SYNONYM DIRECTORY ); my @EXTERNAL_ARRAY = qw( KETTLE FDW ); my @REPORT_ARRAY = qw( SHOW_VERSION SHOW_REPORT SHOW_SCHEMA SHOW_TABLE SHOW_COLUMN SHOW_ENCODING ); my @TEST_ARRAY = qw( TEST TEST_COUNT TEST_VIEW TEST_DATA); my @SOURCES_ARRAY = qw( PACKAGE VIEW TRIGGER FUNCTION PROCEDURE PARTITION TYPE MVIEW ); my @DATA_ARRAY = qw( INSERT COPY); my @CAPABILITIES = qw( QUERY LOAD SCRIPT ); my @MYSQL_SCHEMA_ARRAY = qw( TABLE VIEW GRANT TRIGGER FUNCTION PROCEDURE PARTITION DBLINK ); my @MYSQL_SOURCES_ARRAY = qw( VIEW TRIGGER FUNCTION PROCEDURE PARTITION ); my @MSSQL_SCHEMA_ARRAY = qw( SEQUENCE SEQUENCE_VALUES TABLE PACKAGE VIEW GRANT TRIGGER FUNCTION PROCEDURE TABLESPACE PARTITION TYPE MVIEW DBLINK SYNONYM DIRECTORY ); my @MSSQL_SOURCES_ARRAY = qw( PACKAGE VIEW TRIGGER FUNCTION PROCEDURE PARTITION TYPE MVIEW ); my @GRANT_OBJECTS_ARRAY = ('USER','TABLE','VIEW','MATERIALIZED VIEW','SEQUENCE','PROCEDURE','FUNCTION','PACKAGE BODY','TYPE','SYNONYM','DIRECTORY'); my $TMP_DIR = File::Spec->tmpdir() || '/tmp'; my @OPTIONS = (); # Collect command line arguments GetOptions ( 'a|allow=s' => \$ALLOW, 'b|basedir=s' => \$OUTDIR, 'c|conf=s' => \$FILE_CONF, 'C|cdc_file=s' => \$CDC_FILE, 'd|debug!' => \$DEBUG, 'D|data_type=s' => \$DATA_TYPE, 'e|exclude=s' => \$EXCLUDE, 'g|grant_object=s' => \$GRANT_OBJECT, 'h|help!' => \$HELP, 'i|input_file=s' => \$INPUT_FILE, 'j|jobs=i' => \$THREAD_COUNT, 'J|copies=i' => \$ORACLE_COPIES, 'l|log=s' => \$LOGFILE, 'L|limit=i' => \$DATA_LIMIT, 'm|mysql!' => \$IS_MYSQL, 'M|mssql!' => \$IS_MSSQL, 'n|namespace=s' => \$SCHEMA, 'N|pg_schema=s' => \$PG_SCHEMA, 'o|out=s' => \$OUTFILE, 'O|option=s' => \@OPTIONS, 'p|plsql!' => \$PLSQL, 'P|parallel=i' =>\$PARALLEL_TABLES, 'q|quiet!' => \$QUIET, 'r|relative!' => \$RELATIVE_PATH, 's|source=s' => \$DSN, 'S|scn=s' => \$START_SCN, 't|type=s' => \$EXPORT_TYPE, 'T|temp_dir=s' => \$TMP_DIR, 'u|user=s' => \$DBUSER, 'v|version!' => \$SHOW_VER, 'w|password=s' => \$DBPWD, 'W|where=s' => \@WHERE_CLAUSE, 'x|xtable=s' => \$TABLEONLY, # Obsolete 'forceowner=s' => \$FORCEOWNER, 'nls_lang=s' => \$ORA_ENCODING, 'client_encoding=s' => \$PG_ENCODING, 'view_as_table=s' => \$VIEW_AS_TABLE, 'mview_as_table=s' => \$MVIEW_AS_TABLE, 'estimate_cost!' =>\$ESTIMATE_COST, 'cost_unit_value=i' =>\$COST_UNIT_VALUE, 'dump_as_html!' =>\$DUMP_AS_HTML, 'dump_as_csv!' =>\$DUMP_AS_CSV, 'dump_as_json!' =>\$DUMP_AS_JSON, 'dump_as_sheet!' =>\$DUMP_AS_SHEET, 'dump_as_file_prefix=s' =>\$DUMP_AS_FILE_PREFIX, 'init_project=s' => \$CREATE_PROJECT, 'project_base=s' => \$PROJECT_BASE, 'print_header!' => \$PRINT_HEADER, 'human_days_limit=i' => \$HUMAN_DAY_LIMIT, 'audit_user=s' => \$AUDIT_USER, 'pg_dsn=s' => \$PG_DSN, 'pg_user=s' => \$PG_USER, 'pg_pwd=s' => \$PG_PWD, 'count_rows!' => \$COUNT_ROWS, 'no_header!' => \$NO_HEADER, 'oracle_speed!' => \$ORACLE_SPEED, 'ora2pg_speed!' => \$ORA2PG_SPEED, 'blob_to_lo!' => \$BLOB_TO_LO, 'cdc_ready!' => \$CDC_READY, 'lo_import!' => \$IMPORT_LO, 'drop_if_exists!' => \$DROP_IF_EXISTS, 'delete=s' => \@DELETE_CLAUSE, 'oracle_fdw_prefetch=i' => \$ORACLE_FDW_PREFETCH, 'no_start_scn!' => \$NO_START_SCN, 'no_clean_comment!' => \$NO_CLEAN_COMMENT, ); # Check command line parameters if ($SHOW_VER) { print "Ora2Pg v$VERSION\n"; exit 0; } if ($HELP) { &usage(); } if ($IS_MYSQL) { @SCHEMA_ARRAY = @MYSQL_SCHEMA_ARRAY; @SOURCES_ARRAY = @MYSQL_SOURCES_ARRAY; @EXTERNAL_ARRAY = (); } elsif ($IS_MSSQL) { @SCHEMA_ARRAY = @MSSQL_SCHEMA_ARRAY; @SOURCES_ARRAY = @MSSQL_SOURCES_ARRAY; @EXTERNAL_ARRAY = (); } # Create project repository and useful stuff if ($CREATE_PROJECT) { if (!-d "$PROJECT_BASE") { print "FATAL: Project base directory does not exists: $PROJECT_BASE\n"; &usage(); } print STDERR "Creating project $CREATE_PROJECT.\n"; &create_project($CREATE_PROJECT, $PROJECT_BASE); exit 0; } if ($GRANT_OBJECT && !grep(/^$GRANT_OBJECT$/, @GRANT_OBJECTS_ARRAY)) { print "FATAL: invalid grant object type in -g option. See GRAND_OBJECT configuration directive.\n"; exit 1; } # Clean temporary files unless(opendir(DIR, "$TMP_DIR")) { print "FATAL: can't opendir $TMP_DIR: $!\n"; exit 1; } my @files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); closedir DIR; foreach (@files) { if (-e "$TMP_DIR/$_" and not unlink("$TMP_DIR/$_")) { print "FATAL: can not remove old temporary files $TMP_DIR/$_\n"; exit 1; } } # Check configuration file my $GOES_WITH_DEFAULT = 0; if ($FILE_CONF && ! -e $FILE_CONF) { print "FATAL: can't find configuration file $FILE_CONF\n"; &usage(); } elsif (!$FILE_CONF && ! -e $CONFIG_FILE) { # At least we need configuration to connect to Oracle if (!$DSN || (!$DBUSER && !$ENV{ORA2PG_USER}) || (!$DBPWD && !$ENV{ORA2PG_PASSWD})) { print "FATAL: can't find configuration file $CONFIG_FILE\n"; &usage(); } $CONFIG_FILE = ''; $GOES_WITH_DEFAULT = 1; } push(@CAPABILITIES, @SCHEMA_ARRAY, @REPORT_ARRAY, @DATA_ARRAY, @EXTERNAL_ARRAY, @TEST_ARRAY); # Validate export type $EXPORT_TYPE = uc($EXPORT_TYPE); $EXPORT_TYPE =~ s/^DATA$/COPY/; foreach my $t (split(/[,;\s\t]+/, $EXPORT_TYPE)) { if ($t && !grep(/^$t$/, @CAPABILITIES)) { print "FATAL: Unknown export type: $t. Type supported: ", join(',', @CAPABILITIES), "\n"; &usage(); } } # Preserve barckward compatibility if ($TABLEONLY) { warn "-x | --xtable is deprecated, use -a | --allow option instead.\n"; if (!$ALLOW) { $ALLOW = $TABLEONLY; } } if ($BLOB_TO_LO && !$CREATE_PROJECT && !grep(/^$EXPORT_TYPE$/, 'TABLE', 'INSERT', 'SHOW_COLUMN')) { print "FATAL: option --blob_to_lo can only be used with the INSERT action.\n"; &usage(); } sub getout { my $sig = shift; print STDERR "Received terminating signal ($sig).\n"; $SIG{INT} = \&getout; $SIG{TERM} = \&getout; # Cleaning temporary files unless(opendir(DIR, "$TMP_DIR")) { print "FATAL: can't opendir $TMP_DIR: $!\n"; exit 1; } my @files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); closedir DIR; foreach (@files) { unlink("$TMP_DIR/$_\n"); } exit 1; } $SIG{INT} = \&getout; $SIG{TERM} = \&getout; # Replace ; or space by comma in the user list $AUDIT_USER =~ s/[;\s]+/,/g; # Look if the ALLOW and EXCLUDE values are existing files, # in this case import the content as a space separated list. if (-e $ALLOW) { my $fh = new IO::File; $fh->open($ALLOW) or die "FATAL: can't read file $ALLOW, $!\n"; $ALLOW = ''; while (my $l = <$fh>) { chomp($l); $ALLOW .= "$l "; } $fh->close; $ALLOW =~ s/ $//; } if (-e $EXCLUDE) { my $fh = new IO::File; $fh->open($EXCLUDE) or die "FATAL: can't read file $EXCLUDE, $!\n"; $EXCLUDE = ''; while (my $l = <$fh>) { chomp($l); $EXCLUDE .= "$l "; } $fh->close; $EXCLUDE =~ s/ $//; } # Create an instance of the Ora2Pg perl module my $schema = new Ora2Pg ( config => $FILE_CONF || $CONFIG_FILE, type => $EXPORT_TYPE, debug => $DEBUG, logfile=> $LOGFILE, output => $OUTFILE, output_dir => $OUTDIR, plsql_pgsql => $PLSQL, datasource => $DSN, user => $DBUSER || $ENV{ORA2PG_USER}, password => $DBPWD || $ENV{ORA2PG_PASSWD}, schema => $SCHEMA, pg_schema => $PG_SCHEMA, force_owner => $FORCEOWNER, nls_lang => $ORA_ENCODING, client_encoding => $PG_ENCODING, input_file => $INPUT_FILE, quiet => $QUIET, exclude => $EXCLUDE, allow => $ALLOW, view_as_table => $VIEW_AS_TABLE, mview_as_table => $MVIEW_AS_TABLE, estimate_cost => $ESTIMATE_COST, cost_unit_value => $COST_UNIT_VALUE, dump_as_html => $DUMP_AS_HTML, dump_as_csv => $DUMP_AS_CSV, dump_as_json => $DUMP_AS_JSON, dump_as_sheet => $DUMP_AS_SHEET, dump_as_file_prefix => $DUMP_AS_FILE_PREFIX, thread_count => $THREAD_COUNT, oracle_copies => $ORACLE_COPIES, data_limit => $DATA_LIMIT, parallel_tables => $PARALLEL_TABLES, print_header => $PRINT_HEADER, human_days_limit => $HUMAN_DAY_LIMIT, is_mysql => $IS_MYSQL, is_mssql => $IS_MSSQL, audit_user => $AUDIT_USER, temp_dir => $TMP_DIR, pg_dsn => $PG_DSN, pg_user => $PG_USER, pg_pwd => $PG_PWD, count_rows => $COUNT_ROWS, data_type => $DATA_TYPE, grant_object => $GRANT_OBJECT, no_header => $NO_HEADER, oracle_speed => $ORACLE_SPEED, ora2pg_speed => $ORA2PG_SPEED, psql_relative_path => $RELATIVE_PATH, where => join(' ', @WHERE_CLAUSE), blob_to_lo => $BLOB_TO_LO, start_scn => $START_SCN, cdc_ready => $CDC_READY, lo_import => $IMPORT_LO, cdc_file => $CDC_FILE, drop_if_exists => $DROP_IF_EXISTS, delete => join(' ', @DELETE_CLAUSE), oracle_fdw_prefetch => $ORACLE_FDW_PREFETCH, no_start_scn => $NO_START_SCN, options => join('|', @OPTIONS), no_clean_comment => $NO_CLEAN_COMMENT, ); # Look at configuration file if an input file is defined if (!$INPUT_FILE && !$GOES_WITH_DEFAULT) { my $cf_file = $FILE_CONF || $CONFIG_FILE; my $fh = new IO::File; $fh->open($cf_file) or die "FATAL: can't read configuration file $cf_file, $!\n"; while (my $l = <$fh>) { chomp($l); $l =~ s/\r//gs; $l =~ s/^\s*\#.*$//g; next if (!$l || ($l =~ /^\s+$/)); $l =~ s/^\s*//; $l =~ s/\s*$//; my ($var, $val) = split(/\s+/, $l, 2); $var = uc($var); if ($var eq 'INPUT_FILE' && $val) { $INPUT_FILE = $val; } } $fh->close(); } # Proceed to Oracle DB extraction following # configuration file definitions. if ( ($EXPORT_TYPE !~ /^SHOW_/i) && !$INPUT_FILE ) { $schema->export_schema(); } # Check if error occurs during data export unless(opendir(DIR, "$TMP_DIR")) { print "FATAL: can't opendir $TMP_DIR: $!\n"; exit 1; } @files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); closedir DIR; if ($#files >= 0) { print STDERR "\nWARNING: an error occurs during data export. Please check what's happen.\n\n"; exit 2; } exit(0); #### # Show usage #### sub usage { print qq{ Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] -a | --allow str : Comma separated list of objects to allow from export. Can be used with SHOW_COLUMN too. -b | --basedir dir: Set the default output directory, where files resulting from exports will be stored. -c | --conf file : Set an alternate configuration file other than the default /etc/ora2pg/ora2pg.conf. -C | --cdc_file file: File used to store/read SCN per table during export. default: TABLES_SCN.log in the current directory. This is the file written by the --cdc_ready option. -d | --debug : Enable verbose output. -D | --data_type str : Allow custom type replacement at command line. -e | --exclude str: Comma separated list of objects to exclude from export. Can be used with SHOW_COLUMN too. -h | --help : Print this short help. -g | --grant_object type : Extract privilege from the given object type. See possible values with GRANT_OBJECT configuration. -i | --input file : File containing Oracle PL/SQL code to convert with no Oracle database connection initiated. -j | --jobs num : Number of parallel process to send data to PostgreSQL. -J | --copies num : Number of parallel connections to extract data from Oracle. -l | --log file : Set a log file. Default is stdout. -L | --limit num : Number of tuples extracted from Oracle and stored in memory before writing, default: 10000. -m | --mysql : Export a MySQL database instead of an Oracle schema. -M | --mssql : Export a Microsoft SQL Server database. -n | --namespace schema : Set the Oracle schema to extract from. -N | --pg_schema schema : Set PostgreSQL's search_path. -o | --out file : Set the path to the output file where SQL will be written. Default: output.sql in running directory. -O | --options : Used to override any configuration parameter, it can be used multiple time. Syntax: -O "PARAM_NAME=value" -p | --plsql : Enable PLSQL to PLPGSQL code conversion. -P | --parallel num: Number of parallel tables to extract at the same time. -q | --quiet : Disable progress bar. -r | --relative : use \\ir instead of \\i in the psql scripts generated. -s | --source DSN : Allow to set the Oracle DBI datasource. -S | --scn SCN : Allow to set the Oracle System Change Number (SCN) to use to export data. It will be used in the WHERE clause to get the data. It is used with action COPY or INSERT. -t | --type export: Set the export type. It will override the one given in the configuration file (TYPE). -T | --temp_dir dir: Set a distinct temporary directory when two or more ora2pg are run in parallel. -u | --user name : Set the Oracle database connection user. ORA2PG_USER environment variable can be used instead. -v | --version : Show Ora2Pg Version and exit. -w | --password pwd : Set the password of the Oracle database user. ORA2PG_PASSWD environment variable can be used instead. -W | --where clause : Set the WHERE clause to apply to the Oracle query to retrieve data. Can be used multiple time. --forceowner : Force ora2pg to set tables and sequences owner like in Oracle database. If the value is set to a username this one will be used as the objects owner. By default it's the user used to connect to the Pg database that will be the owner. --nls_lang code: Set the Oracle NLS_LANG client encoding. --client_encoding code: Set the PostgreSQL client encoding. --view_as_table str: Comma separated list of views to export as table. --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT --cost_unit_value minutes: Number of minutes for a cost evaluation unit. default: 5 minutes, corresponds to a migration conducted by a PostgreSQL expert. Set it to 10 if this is your first migration. --dump_as_html : Force ora2pg to dump report in HTML, used only with SHOW_REPORT. Default is to dump report as simple text. --dump_as_csv : As above but force ora2pg to dump report in CSV. --dump_as_json : As above but force ora2pg to dump report in JSON. --dump_as_sheet : Report migration assessment with one CSV line per database. --dump_as_file_prefix : Filename prefix, suffix will be added depending on dump_as_* selected switches, suffixes will be .html, .csv, .json. --init_project name: Initialise a typical ora2pg project tree. Top directory will be created under project base dir. --project_base dir : Define the base dir for ora2pg project trees. Default is current directory. --print_header : Used with --dump_as_sheet to print the CSV header especially for the first run of ora2pg. --human_days_limit num : Set the number of human-days limit where the migration assessment level switch from B to C. Default is set to 5 human-days. --audit_user list : Comma separated list of usernames to filter queries in the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT and QUERY export type. --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. --pg_user name : Set the PostgreSQL user to use. --pg_pwd password : Set the PostgreSQL password to use. --count_rows : Force ora2pg to perform a real row count in TEST, TEST_COUNT and SHOW_TABLE actions. --no_header : Do not append Ora2Pg header to output file --oracle_speed : Use to know at which speed Oracle is able to send data. No data will be processed or written. --ora2pg_speed : Use to know at which speed Ora2Pg is able to send transformed data. Nothing will be written. --blob_to_lo : export BLOB as large objects, can only be used with action SHOW_COLUMN, TABLE and INSERT. --cdc_ready : use current SCN per table to export data and register them into a file named TABLES_SCN.log per default. It can be changed using -C | --cdc_file. --lo_import : use psql \\lo_import command to import BLOB as large object. Can be use to import data with COPY and import large object manually in a second pass. It is recquired for BLOB > 1GB. See documentation for more explanation. --mview_as_table str: Comma separated list of materialized views to export as regular table. --drop_if_exists : Drop the object before creation if it exists. --delete clause : Set the DELETE clause to apply to the Oracle query to be applied before importing data. Can be used multiple time. --oracle_fdw_prefetch: Set the oracle_fdw prefetch value. Larger values generally result in faster data transfer at the cost of greater memory utilisation at the destination. --no_start_scn : Force Ora2Pg to not use a SCN to export data. By default the current SCN is used to export data from all tables. --no_clean_comment: do not try to remove comments in source file before parsing. In some cases it could table a very long time. See full documentation at https://ora2pg.darold.net/ for more help or see manpage with 'man ora2pg'. ora2pg will return 0 on success, 1 on error. It will return 2 when a child process has been interrupted and you've gotten the warning message: "WARNING: an error occurs during data export. Please check what's happen." Most of the time this is an OOM issue, first try reducing DATA_LIMIT value. }; exit 1; } #### # Create a generic project tree #### sub create_project { my ($create_project, $project_base) = @_; # Look at default configuration file to use my $conf_file = $CONFIG_FILE . '.dist'; if ($FILE_CONF) { # Use file given in parameter $conf_file = $FILE_CONF; } if (!-f $conf_file || -z $conf_file) { print "FATAL: file $conf_file does not exists.\n"; exit 1; } # Build entire project tree my $base_path = $project_base . '/' . $create_project; if (-e $base_path) { print "FATAL: project directory exists $base_path\n"; exit 1; } mkdir("$base_path"); print "$base_path/\n"; mkdir("$base_path/schema"); print "\tschema/\n"; foreach my $exp (sort @SCHEMA_ARRAY ) { my $tpath = lc($exp); $tpath =~ s/y$/ie/; if ($exp ne 'SEQUENCE_VALUES') { mkdir("$base_path/schema/" . $tpath . 's'); print "\t\t" . $tpath . "s/\n"; } else { mkdir("$base_path/schema/" . $tpath); print "\t\t" . $tpath . "/\n"; } } mkdir("$base_path/sources"); print "\tsources/\n"; foreach my $exp (sort @SOURCES_ARRAY ) { my $tpath = lc($exp); $tpath =~ s/y$/ie/; mkdir("$base_path/sources/" . $tpath . 's'); print "\t\t" . $tpath . "s/\n"; } mkdir("$base_path/data"); print "\tdata/\n"; mkdir("$base_path/config"); print "\tconfig/\n"; mkdir("$base_path/reports"); print "\treports/\n"; print "\n"; # Copy configuration file and transform it as a generic one print "Generating generic configuration file\n"; if (open(IN, "$conf_file")) { my @cf = ; close(IN); # Create a generic configuration file only if it has the .dist extension # otherwise use the configuration given at command line (-c option) if ($conf_file =~ /\.dist/) { &make_config_generic(\@cf); } unless(open(OUT, ">$base_path/config/ora2pg.conf")) { print "FATAL: can't write to file $base_path/config/ora2pg.conf\n"; exit 1; } print OUT @cf; close(OUT); } else { print "FATAL: can not read file $conf_file, $!.\n"; exit 1; } # Generate shell script to execute all export print "Creating script export_schema.sh to automate all exports.\n"; unless(open(OUT, "> $base_path/export_schema.sh")) { print "FATAL: Can't write to file $base_path/export_schema.sh\n"; exit 1; } my $ablob_to_lo = ''; my $data_action = 'COPY'; if ($BLOB_TO_LO) { $ablob_to_lo = '--blob_to_lo'; $data_action = 'INSERT'; } print OUT qq{#!/bin/sh #------------------------------------------------------------------------------- # # Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION # #------------------------------------------------------------------------------- }; print OUT "EXPORT_TYPE=\"", join(' ', @SCHEMA_ARRAY), "\"\n"; print OUT "SOURCE_TYPE=\"", join(' ', @SOURCES_ARRAY), "\"\n"; print OUT "namespace=\".\"\n"; print OUT "unit_cost=$COST_UNIT_VALUE\n"; print OUT qq{ ora2pg -t SHOW_TABLE -c \$namespace/config/ora2pg.conf > \$namespace/reports/tables.txt ora2pg -t SHOW_COLUMN -c \$namespace/config/ora2pg.conf $ablob_to_lo > \$namespace/reports/columns.txt ora2pg -t SHOW_REPORT -c \$namespace/config/ora2pg.conf --dump_as_html --cost_unit_value \$unit_cost --estimate_cost > \$namespace/reports/report.html for etype in \$(echo \$EXPORT_TYPE | tr " " "\\n") do ltype=`echo \$etype | tr '[:upper:]' '[:lower:]'` ltype=`echo \$ltype | sed 's/y\$/ie/'` ltype=`echo \$ltype | sed 's/s\$//'` if [ "\$etype" = "TABLE" ]; then blob_to_lo="$ablob_to_lo" else blob_to_lo="" fi echo "Running: ora2pg -p -t \$etype -o \$ltype.sql -b \$namespace/schema/\$\{ltype\}s -c \$namespace/config/ora2pg.conf \$\{blob_to_lo\}" ora2pg -p -t \$etype -o \$ltype.sql -b \$namespace/schema/\$\{ltype\}s -c \$namespace/config/ora2pg.conf \$\{blob_to_lo\} ret=`grep "Nothing found" \$namespace/schema/\$\{ltype\}s/\$ltype.sql 2> /dev/null` if [ ! -z "\$ret" ]; then rm \$namespace/schema/\$\{ltype\}s/\$ltype.sql fi done for etype in \$(echo \$SOURCE_TYPE | tr " " "\\n") do ltype=`echo \$etype | tr '[:upper:]' '[:lower:]'` ltype=`echo \$ltype | sed 's/y\$/ie/'` echo "Running: ora2pg -t \$etype -o \$ltype.sql -b \$namespace/sources/\$\{ltype\}s -c \$namespace/config/ora2pg.conf" ora2pg -t \$etype -o \$ltype.sql -b \$namespace/sources/\$\{ltype\}s -c \$namespace/config/ora2pg.conf ret=`grep "Nothing found" \$namespace/sources/\$\{ltype\}s/\$ltype.sql 2> /dev/null` if [ ! -z "\$ret" ]; then rm \$namespace/sources/\$\{ltype\}s/\$ltype.sql fi done echo echo echo "To extract data use the following command:" echo echo "ora2pg -t $data_action -o data.sql -b \$namespace/data -c \$namespace/config/ora2pg.conf $ablob_to_lo" echo exit 0 }; close(OUT); chmod(0700, "$base_path/export_schema.sh"); # On Windows system generate a Powershell script to execute all export if ($^O =~ /MSWin32|dos/i) { print "Creating Powershell script export_schema.ps1 to automate all exports.\n"; unless(open(OUT, "> $base_path/export_schema.ps1")) { print "FATAL: Can't write to file $base_path/export_schema.ps1\n"; exit 1; } print OUT qq{ #------------------------------------------------------------------------------- # # Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION # #------------------------------------------------------------------------------- }; print OUT "\$EXPORT_TYPE=", join(',', map "'$_'",@SCHEMA_ARRAY),"\n"; print OUT "\$SOURCE_TYPE=", join(',', map "'$_'",@SOURCES_ARRAY), "\n"; print OUT "\$namespace=\".\"\n"; print OUT "\$unit_cost=$COST_UNIT_VALUE\n"; print OUT qq{ ora2pg -t SHOW_TABLE -c \$namespace/config/ora2pg.conf > \$namespace/reports/tables.txt ora2pg -t SHOW_COLUMN -c \$namespace/config/ora2pg.conf $ablob_to_lo > \$namespace/reports/columns.txt ora2pg -t SHOW_REPORT -c \$namespace/config/ora2pg.conf --dump_as_html --cost_unit_value \$unit_cost --estimate_cost \> \$namespace/reports/report.html foreach (\$etype in \$EXPORT_TYPE) { \$ltype = \$etype.ToLower() -replace 'y\$', 'ie' \$ltype = \$etype.ToLower() -replace 's\$', '' if ( "\$etype" -eq "TABLE" ) { blob_to_lo="$ablob_to_lo" } else { blob_to_lo="" } \$cmd="ora2pg -p -t \$etype -o \$ltype.sql -b \$namespace/schema/\${ltype}s -c \$namespace/config/ora2pg.conf \$\{blob_to_lo\}" Write-Host "Running: \$cmd" Invoke-Expression \$cmd Select-String -Pattern 'Nothing found' -Path \$namespace/schema/\${ltype}s/\$ltype.sql -List |Remove-Item } foreach (\$etype in \$SOURCE_TYPE) { \$ltype = \$etype.ToLower() -replace 'y\$', 'ie' \$cmd="ora2pg -t \$etype -o \$ltype.sql -b \$namespace/sources/\${ltype}s -c \$namespace/config/ora2pg.conf" write-host "Running: \$cmd" Invoke-Expression \$cmd Select-String -Pattern 'Nothing found' -Path \$namespace/sources/\${ltype}s/\$ltype.sql -List |Remove-Item } Write-Host Write-Host Write-Host "To extract data use the following command:" Write-Host Write-Host "ora2pg -t $data_action -o data.sql -b \$namespace/data -c \$namespace/config/ora2pg.conf $ablob_to_lo" Write-Host exit 0; }; } close(OUT); # Generate shell script to execute all import print "Creating script import_all.sh to automate all imports.\n"; my $exportype = "EXPORT_TYPE=\"TYPE " . join(' ', grep( !/^TYPE$/, @SCHEMA_ARRAY)) . "\"\n"; unless(open(OUT, "> $base_path/import_all.sh")) { print "FATAL: Can't write to file $base_path/import_all.sh\n"; exit 1; } $ablob_to_lo = ''; $data_action = 'COPY'; if ($BLOB_TO_LO) { $ablob_to_lo = '--blob_to_lo'; $data_action = 'INSERT'; } while (my $l = ) { $l =~ s/^EXPORT_TYPE=.*/$exportype/s; $l =~ s/ORA2PG_VERSION/$VERSION/s; $l =~ s/DATA_ACTION/$data_action/s; $l =~ s/BLOB2LO/$ablob_to_lo/s; print OUT $l; } close(OUT); chmod(0700, "$base_path/import_all.sh"); } #### # Set a generic configuration #### sub make_config_generic { my $conf_arr = shift; chomp(@$conf_arr); my $schema = 'CHANGE_THIS_SCHEMA_NAME'; $schema = $SCHEMA if ($SCHEMA); for (my $i = 0; $i <= $#{$conf_arr}; $i++) { if ($IS_MYSQL) { $conf_arr->[$i] =~ s/^# Set Oracle database/# Set MySQL database/; $conf_arr->[$i] =~ s/^(ORACLE_DSN.*dbi):Oracle:(.*);sid=SIDNAME/$1:mysql:$2;database=dbname/; $conf_arr->[$i] =~ s/CHANGE_THIS_SCHEMA_NAME/CHANGE_THIS_DB_NAME/; $conf_arr->[$i] =~ s/#REPLACE_ZERO_DATE.*/REPLACE_ZERO_DATE\t-INFINITY/; } elsif ($IS_MSSQL) { $conf_arr->[$i] =~ s/^# Set Oracle database/# Set MSSQL database/; $conf_arr->[$i] =~ s/^(ORACLE_DSN.*dbi):Oracle:(.*);sid=SIDNAME/$1:ODBC:driver=msodbcsql18;$2;database=dbname/; $conf_arr->[$i] =~ s/;host=/;server=/; $conf_arr->[$i] =~ s/;port=1521/;port=1433;TrustCertificate=yes;TrustServerCertificate=Yes/; } elsif ($ENV{ORACLE_HOME}) { $conf_arr->[$i] =~ s/^ORACLE_HOME.*/ORACLE_HOME\t$ENV{ORACLE_HOME}/; } $conf_arr->[$i] =~ s/^USER_GRANTS.*0/USER_GRANTS\t1/; $conf_arr->[$i] =~ s/^#SCHEMA.*SCHEMA_NAME/SCHEMA\t$schema/; $conf_arr->[$i] =~ s/^(BINMODE.*)/#$1/; $conf_arr->[$i] =~ s/^PLSQL_PGSQL.*1/PLSQL_PGSQL\t0/; $conf_arr->[$i] =~ s/^FILE_PER_CONSTRAINT.*0/FILE_PER_CONSTRAINT\t1/; $conf_arr->[$i] =~ s/^FILE_PER_INDEX.*0/FILE_PER_INDEX\t1/; $conf_arr->[$i] =~ s/^FILE_PER_FKEYS.*0/FILE_PER_FKEYS\t1/; $conf_arr->[$i] =~ s/^FILE_PER_TABLE.*0/FILE_PER_TABLE\t1/; $conf_arr->[$i] =~ s/^FILE_PER_FUNCTION.*0/FILE_PER_FUNCTION\t1/; $conf_arr->[$i] =~ s/^TRUNCATE_TABLE.*0/TRUNCATE_TABLE\t1/; $conf_arr->[$i] =~ s/^DISABLE_SEQUENCE.*0/DISABLE_SEQUENCE\t1/; $conf_arr->[$i] =~ s/^DISABLE_TRIGGERS.*0/DISABLE_TRIGGERS\t1/; $conf_arr->[$i] =~ s/^(CLIENT_ENCODING.*)/#$1/; $conf_arr->[$i] =~ s/^(NLS_LANG.*)/#$1/; $conf_arr->[$i] =~ s/^#LONGREADLEN.*1047552/LONGREADLEN\t1047552/; $conf_arr->[$i] =~ s/^AUTODETECT_SPATIAL_TYPE.*0/AUTODETECT_SPATIAL_TYPE\t1/; $conf_arr->[$i] =~ s/^NO_LOB_LOCATOR.*/NO_LOB_LOCATOR\t0/; $conf_arr->[$i] =~ s/^USE_LOB_LOCATOR.*/USE_LOB_LOCATOR\t1/; $conf_arr->[$i] =~ s/^FTS_INDEX_ONLY.*0/FTS_INDEX_ONLY\t1/; $conf_arr->[$i] =~ s/^DISABLE_UNLOGGED.*0/DISABLE_UNLOGGED\t1/; $conf_arr->[$i] =~ s/^EMPTY_LOB_NULL.*0/EMPTY_LOB_NULL\t1/; $conf_arr->[$i] =~ s/^PG_NUMERIC_TYPE.*1/PG_NUMERIC_TYPE\t0/; $conf_arr->[$i] =~ s/^NULL_EQUAL_EMPTY.*0/NULL_EQUAL_EMPTY\t1/; if ($DSN) { $conf_arr->[$i] =~ s/^ORACLE_DSN.*/ORACLE_DSN\t$DSN/; } if ($DBUSER) { $conf_arr->[$i] =~ s/^ORACLE_USER.*/ORACLE_USER\t$DBUSER/; } if ($DBPWD) { $conf_arr->[$i] =~ s/^ORACLE_PWD.*/ORACLE_PWD\t$DBPWD/; } } map { s/$/\n/; } @$conf_arr; } __DATA__ #!/bin/sh #------------------------------------------------------------------------------- # # Script used to load exported sql files into PostgreSQL in practical manner # allowing you to chain and automatically import schema and data. # # Generated by Ora2Pg, the Oracle database Schema converter, version ORA2PG_VERSION # #------------------------------------------------------------------------------- EXPORT_TYPE="TYPE,TABLE,PARTITION,VIEW,MVIEW,FUNCTION,PROCEDURE,SEQUENCE,TRIGGER,SYNONYM,DIRECTORY,DBLINK" AUTORUN=0 NAMESPACE=. NO_CONSTRAINTS=0 IMPORT_INDEXES_AFTER=0 DEBUG=0 IMPORT_SCHEMA=0 IMPORT_DATA=0 IMPORT_CONSTRAINTS=0 NO_DBCHECK=0 # Message functions die() { echo "ERROR: $1" 1>&2 exit 1 } usage() { echo "usage: `basename $0` [options]" echo "" echo "Script used to load exported sql files into PostgreSQL in practical manner" echo "allowing you to chain and automatically import schema and data." echo "" echo "options:" echo " -a import data only" echo " -b filename SQL script to execute just after table creation to fix database schema" echo " -d dbname database name for import" echo " -D enable debug mode, will only show what will be done" echo " -e encoding database encoding to use at creation (default: UTF8)" echo " -f force no check of user and database existing and do not try to create them" echo " -h hostname hostname of the PostgreSQL server (default: unix socket)" echo " -i only load indexes, constraints and triggers" echo " -I do not try to load indexes, constraints and triggers" echo " -j cores number of connection to use to import data or indexes into PostgreSQL" echo " -n schema comma separated list of schema to create" echo " -o username owner of the database to create" echo " -p port listening port of the PostgreSQL server (default: 5432)" echo " -P cores number of tables to process at same time for data import" echo " -s import schema only, do not try to import data" echo " -t export comma separated list of export type to import (same as ora2pg)" echo " -U username username to connect to PostgreSQL (default: peer username)" echo " -x import indexes and constraints after data" echo " -y reply Yes to all questions for automatic import" echo echo " -? print help" echo exit $1 } # Function to emulate Perl prompt function confirm () { msg=$1 if [ "$AUTORUN" != "0" ]; then true else if [ -z "$msg" ]; then msg="Are you sure? [y/N/q]" fi # call with a prompt string or use a default read -r -p "${msg} [y/N/q] " response case $response in [yY][eE][sS]|[yY]) true ;; [qQ][uU][iI][tT]|[qQ]) exit ;; *) false ;; esac fi } # Function used to import constraints and indexes import_constraints () { if [ -r "$NAMESPACE/schema/tables/INDEXES_table.sql" ]; then if confirm "Would you like to import indexes from $NAMESPACE/schema/tables/INDEXES_table.sql?" ; then if [ -z "$IMPORT_JOBS" ]; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/INDEXES_table.sql" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/INDEXES_table.sql if [ $? -ne 0 ]; then die "can not import indexes." fi fi else echo "Running: ora2pg -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/INDEXES_table.sql" if [ $DEBUG -eq 0 ]; then ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/INDEXES_table.sql if [ $? -ne 0 ]; then die "can not import indexes." fi fi fi fi fi if [ -r "$NAMESPACE/schema/tables/CONSTRAINTS_table.sql" ]; then if confirm "Would you like to import constraints from $NAMESPACE/schema/tables/CONSTRAINTS_table.sql?" ; then if [ -z "$IMPORT_JOBS" ]; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/CONSTRAINTS_table.sql" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/CONSTRAINTS_table.sql if [ $? -ne 0 ]; then die "can not import constraints." fi fi else echo "Running: ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/CONSTRAINTS_table.sql" if [ $DEBUG -eq 0 ]; then ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/CONSTRAINTS_table.sql if [ $? -ne 0 ]; then die "can not import constraints." fi fi fi fi fi if [ -r "$NAMESPACE/schema/tables/FKEYS_table.sql" ]; then if confirm "Would you like to import foreign keys from $NAMESPACE/schema/tables/FKEYS_table.sql?" ; then if [ -z "$IMPORT_JOBS" ]; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FKEYS_table.sql" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FKEYS_table.sql if [ $? -ne 0 ]; then die "can not import foreign keys." fi fi else echo "Running: ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FKEYS_table.sql" if [ $DEBUG -eq 0 ]; then ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FKEYS_table.sql if [ $? -ne 0 ]; then die "can not import foreign keys." fi fi fi fi fi if [ -r "$NAMESPACE/schema/tables/FTS_INDEXES_table.sql" ]; then if confirm "Would you like to import FTS indexes from $NAMESPACE/schema/tables/FTS_INDEXES_table.sql?" ; then if [ -z "$IMPORT_JOBS" ]; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FTS_INDEXES_table.sql" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FTS_INDEXES_table.sql if [ $? -ne 0 ]; then die "can not import FTS indexes." fi fi else echo "Running: ora2pg -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FTS_INDEXES_table.sql" if [ $DEBUG -eq 0 ]; then ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FTS_INDEXES_table.sql if [ $? -ne 0 ]; then die "can not import FTS indexes." fi fi fi fi fi if [ -r "$NAMESPACE/schema/triggers/trigger.sql" ]; then if confirm "Would you like to import TRIGGER from $NAMESPACE/schema/triggers/trigger.sql?" ; then echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/triggers/trigger.sql" if [ $DEBUG -eq 0 ]; then psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/triggers/trigger.sql if [ $? -ne 0 ]; then die "an error occurs when importing file $NAMESPACE/schema/triggers/trigger.sql." fi fi fi fi } # Command line options while getopts "b:d:e:h:j:l:n:o:p:P:t:U:aDfiIsyx?" opt; do case "$opt" in a) IMPORT_DATA=1;; b) SQL_POST_SCRIPT=$OPTARG;; d) DB_NAME=$OPTARG;; D) DEBUG=1;; e) DB_ENCODING=" -E $OPTARG";; f) NO_DBCHECK=1;; h) DB_HOST=" -h $OPTARG";; i) IMPORT_CONSTRAINTS=1;; I) NO_CONSTRAINTS=1;; j) IMPORT_JOBS=" -j $OPTARG";; n) DB_SCHEMA=$OPTARG;; o) DB_OWNER=$OPTARG;; p) DB_PORT=" -p $OPTARG";; P) PARALLEL_TABLES=" -P $OPTARG";; s) IMPORT_SCHEMA=1;; t) EXPORT_TYPE=$OPTARG;; U) DB_USER=" -U $OPTARG";; x) IMPORT_INDEXES_AFTER=1;; y) AUTORUN=1;; "?") usage 1;; *) die "Unknown error while processing options";; esac done # Check if post tables import SQL script is readable if [ ! -z "$SQL_POST_SCRIPT" ]; then if [ ! -r "$SQL_POST_SCRIPT" ]; then die "the SQL script $SQL_POST_SCRIPT is not readable." fi fi # A database name is mandatory if [ -z "$DB_NAME" ]; then die "you must give a PostgreSQL database name (see -d option)." fi # A database owner is mandatory if [ -z "$DB_OWNER" ]; then die "you must give a username to be used as owner of database (see -o option)." fi # Check if the project directory is readable if [ ! -r "$NAMESPACE/schema/tables/table.sql" ]; then die "project directory '$NAMESPACE' is not valid or is not readable." fi # If constraints and indexes files are present propose to import these objects if [ $IMPORT_CONSTRAINTS -eq 1 ]; then if confirm "Would you like to load indexes, constraints and triggers?" ; then import_constraints fi exit 0 fi # When a PostgreSQL schema list is provided, create them if [ $IMPORT_DATA -eq 0 ]; then is_superuser='f' if [ $NO_DBCHECK -eq 0 ]; then # Create owner user user_exists=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select usename from pg_user where usename='$DB_OWNER';" 2>/dev/null` is_superuser=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select usesuper from pg_user where usename='$DB_OWNER';" 2>/dev/null`; if [ "a$user_exists" = "a" ]; then if confirm "Would you like to create the owner of the database $DB_OWNER?" ; then echo "Running: createuser$DB_HOST$DB_PORT$DB_USER --no-superuser --no-createrole --no-createdb $DB_OWNER" if [ $DEBUG -eq 0 ]; then createuser$DB_HOST$DB_PORT$DB_USER --no-superuser --no-createrole --no-createdb $DB_OWNER if [ $? -ne 0 ]; then die "can not create user $DB_OWNER." fi fi fi else echo "Database owner $DB_OWNER already exists, skipping creation." fi # Create database if required if [ "a$DB_ENCODING" = "a" ]; then DB_ENCODING=" -E UTF8" fi db_exists=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select datname from pg_database where datname='$DB_NAME';"` if [ "a$db_exists" = "a" ]; then if confirm "Would you like to create the database $DB_NAME?" ; then echo "Running: createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME" if [ $DEBUG -eq 0 ]; then createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME if [ $? -ne 0 ]; then die "can not create database $DB_NAME." fi fi fi else if confirm "Would you like to drop the database $DB_NAME before recreate it?" ; then echo "Running: dropdb$DB_HOST$DB_PORT$DB_USER $DB_NAME" if [ $DEBUG -eq 0 ]; then dropdb$DB_HOST$DB_PORT$DB_USER $DB_NAME if [ $? -ne 0 ]; then die "can not drop database $DB_NAME." fi fi echo "Running: createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME" if [ $DEBUG -eq 0 ]; then createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME if [ $? -ne 0 ]; then die "can not create database $DB_NAME." fi fi fi fi fi # When schema list is provided, create them if [ "a$DB_SCHEMA" != "a" ]; then nspace_list='' for enspace in $(echo $DB_SCHEMA | tr "," "\n") do lnspace=`echo $enspace | tr '[:upper:]' '[:lower:]'` if confirm "Would you like to create schema $lnspace in database $DB_NAME?" ; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -c \"CREATE SCHEMA $lnspace;\"" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -c "CREATE SCHEMA $lnspace;" if [ $? -ne 0 ]; then die "can not create schema $DB_SCHEMA." fi fi nspace_list="$nspace_list$lnspace," fi done # Change search path of the owner if [ "a$nspace_list" != "a" ]; then if confirm "Would you like to change search_path of the database owner?" ; then echo "Running: psql$DB_HOST$DB_PORT$DB_USER -d $DB_NAME -c \"ALTER ROLE $DB_OWNER SET search_path TO ${nspace_list}public;\"" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT$DB_USER -d $DB_NAME -c "ALTER ROLE $DB_OWNER SET search_path TO ${nspace_list}public;" if [ $? -ne 0 ]; then die "can not change search_path." fi fi fi fi fi # Then import all files from project directory for etype in $(echo $EXPORT_TYPE | tr "," "\n") do if [ $NO_CONSTRAINTS -eq 1 ] && [ $etype = "TRIGGER" ]; then continue fi if [ $etype = "GRANT" ] || [ $etype = "TABLESPACE" ]; then continue fi ltype=`echo $etype | tr '[:upper:]' '[:lower:]'` ltype=`echo $ltype | sed 's/y$/ie/'` ltype=`echo $ltype | sed 's/s$//'` if [ -r "$NAMESPACE/schema/${ltype}s/$ltype.sql" ]; then if confirm "Would you like to import $etype from $NAMESPACE/schema/${ltype}s/$ltype.sql?" ; then echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/${ltype}s/$ltype.sql" if [ $DEBUG -eq 0 ]; then psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/${ltype}s/$ltype.sql if [ $? -ne 0 ]; then die "an error occurs when importing file $NAMESPACE/schema/${ltype}s/$ltype.sql." fi fi fi fi if [ ! -z "$SQL_POST_SCRIPT" ] && [ $etype = "TABLE" ]; then if confirm "Would you like to execute SQL script $SQL_POST_SCRIPT?" ; then echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $SQL_POST_SCRIPT" if [ $DEBUG -eq 0 ]; then psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $SQL_POST_SCRIPT if [ $? -ne 0 ]; then die "an error occurs when importing file $SQL_POST_SCRIPT." fi fi fi fi done # If constraints and indexes files are present propose to import these objects if [ $NO_CONSTRAINTS -eq 0 ] && [ $IMPORT_INDEXES_AFTER -eq 0 ]; then if confirm "Would you like to process indexes and constraints before loading data?" ; then IMPORT_INDEXES_AFTER=0 import_constraints else IMPORT_INDEXES_AFTER=1 fi fi # When the database owner is not superuser use postgres instead q_user='postgres' if [ "$is_superuser" = "t" ]; then q_user=$DB_OWNER fi # Import objects that need superuser privilege: GRANT and TABLESPACE if [ -r "$NAMESPACE/schema/grants/grant.sql" ]; then if confirm "Would you like to import GRANT from $NAMESPACE/schema/grants/grant.sql?" ; then echo "Running: psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/grants/grant.sql" if [ $DEBUG -eq 0 ]; then psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/grants/grant.sql if [ $? -ne 0 ]; then die "an error occurs when importing file $NAMESPACE/schema/grants/grant.sql." fi fi fi fi if [ -r "$NAMESPACE/schema/tablespaces/tablespace.sql" ]; then if confirm "Would you like to import TABLESPACE from $NAMESPACE/schema/tablespaces/tablespace.sql?" ; then echo "Running: psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/tablespaces/tablespace.sql" if [ $DEBUG -eq 0 ]; then psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/tablespaces/tablespace.sql if [ $? -ne 0 ]; then die "an error occurs when importing file $NAMESPACE/schema/tablespaces/tablespace.sql." fi fi fi fi fi # Check if we must just import schema or proceed to data import too if [ $IMPORT_SCHEMA -eq 0 ]; then # set the PostgreSQL datasource pgdsn_defined=`grep "^PG_DSN" config/ora2pg.conf | sed 's/.*dbi:Pg/dbi:Pg/'` if [ "a$pgdsn_defined" = "a" ]; then if [ "a$DB_HOST" != "a" ]; then pgdsn_defined="dbi:Pg:dbname=$DB_NAME;host=$DB_HOST" else #default to unix socket pgdsn_defined="dbi:Pg:dbname=$DB_NAME;" fi if [ "a$DB_PORT" != "a" ]; then pgdsn_defined="$pgdsn_defined;port=$DB_PORT" else pgdsn_defined="$pgdsn_defined;port=5432" fi fi # remove command line option from the DSN string pgdsn_defined=`echo "$pgdsn_defined" | sed 's/ -. //g'` # If data file is present propose to import data if [ -r "$NAMESPACE/data/data.sql" ]; then if confirm "Would you like to import data from $NAMESPACE/data/data.sql?" ; then echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/data/data.sql" if [ $DEBUG -eq 0 ]; then psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/data/data.sql if [ $? -ne 0 ]; then die "an error occurs when importing file $NAMESPACE/data/data.sql." fi fi fi else # Import data directly from PostgreSQL if confirm "Would you like to import data from Oracle database directly into PostgreSQL?" ; then echo "Running: ora2pg$IMPORT_JOBS$PARALLEL_TABLES -c config/ora2pg.conf -t DATA_ACTION --pg_dsn \"$pgdsn_defined\" --pg_user $DB_OWNER BLOB2LO" if [ $DEBUG -eq 0 ]; then ora2pg$IMPORT_JOBS$PARALLEL_TABLES -c config/ora2pg.conf -t DATA_ACTION --pg_dsn "$pgdsn_defined" --pg_user $DB_OWNER BLOB2LO if [ $? -ne 0 ]; then die "an error occurs when importing data." fi fi fi fi if [ $NO_CONSTRAINTS -eq 0 ] && [ $IMPORT_DATA -eq 0 ]; then # Import indexes and constraint after data if [ $IMPORT_INDEXES_AFTER -eq 1 ]; then import_constraints fi fi fi echo -e "\nOra2Pg ending" exit 0 ora2pg-25.0/scripts/ora2pg_scanner000077500000000000000000000251151500113072400171470ustar00rootroot00000000000000#!/usr/bin/perl #------------------------------------------------------------------------------ # Project : Oracle to Postgresql converter # Name : ora2pg_scanner # Author : Gilles Darold, gilles _AT_ darold _DOT_ net # Copyright: Copyright (c) 2000-2025 : Gilles Darold - All rights reserved - # Function : Script used to scan a list of DSN and generate reports # Usage : ora2pg_scanner -l dsn_csv_file -o outdir #------------------------------------------------------------------------------ # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see < http://www.gnu.org/licenses/ >. # #------------------------------------------------------------------------------ use strict; use Getopt::Long qw(:config no_ignore_case bundling); my $VERSION = '25.0'; my @DB_DSN = (); my $OUTDIR = ''; my $DRYRUN = 0; my $INPUT_FILE = ''; my $CONF_FILE = ''; my $HELP = 0; my $ORA2PG_CMD = 'ora2pg'; my $BINPATH = ''; my $SEP = ($^O =~ /MSWin32|dos/i) ? "\\" : "/"; my $COST_UNIT = 5; my $FORMAT = 'html'; # Collect command line arguments GetOptions ( 'c|config=s' => \$CONF_FILE, 'b|binpath=s' => \$BINPATH, 'f|format=s' => \$FORMAT, 'l|list=s' => \$INPUT_FILE, 't|test!' => \$DRYRUN, 'o|outdir=s' => \$OUTDIR, 'u|unit=s' => \$COST_UNIT, 'h|help!' => \$HELP, ); $OUTDIR = 'output' if (!$OUTDIR); if (!$INPUT_FILE || !-e $INPUT_FILE || $HELP) { usage(); } $FORMAT = lc($FORMAT); if ($FORMAT ne 'html' && $FORMAT ne 'json') { die "FATAL: output format for reports must be html or json\n"; } if ($BINPATH) { $BINPATH =~ s/\Q$SEP\E$//; if (-e "$BINPATH$SEP$ORA2PG_CMD") { $ORA2PG_CMD = "$BINPATH$SEP$ORA2PG_CMD"; } else { die "FATAL: path to ora2pg binary must exists: $BINPATH$SEP$ORA2PG_CMD\n"; } } if ($^O =~ /MSWin32|dos/i) { $ORA2PG_CMD = "perl $ORA2PG_CMD"; } open(IN, $INPUT_FILE) or die "FATAL: can not read file $INPUT_FILE, $!\n"; while (my $l = ) { #"type","schema/database","dsn","user","password","audit users" #"MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","mysecret" #"ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager","hr;system;scott" #"MSSQL","HR","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","usrname","pwd" # skip header line chomp($l); $l =~ s/\r//; next if ($l !~ /^["]*(MYSQL|ORACLE|MSSQL)["]*,/i); $l =~ s/"//gs; my @data = split(/,/, $l); if ($#data < 4) { die "FATAL: wrong number of field at line: $l\n"; } my ($type, $schema, $dsn, $user, $passwd, $audit_user) = split(/,/, $l); push(@DB_DSN, { ( 'type' => uc($type), 'schema' => $schema, 'dsn' => $dsn, 'user' => $user, 'pwd' => $passwd, 'audit_user' => $audit_user, 'sid' => '', 'host' => '' ) } ); } close(IN); # Create the output directory if (!$DRYRUN) { if (!-d "$OUTDIR") { mkdir "$OUTDIR"; } else { print "FATAL: output directory already exists, $OUTDIR.\n"; exit 1; } } else { print "Performing connection test only by retrieving the requested schema or database.\n"; } # Start to generate call to ora2pg my $header = ' --print_header'; for (my $i = 0; $i < @DB_DSN; $i++) { $header = '' if ($i > 0); $ENV{ORA2PG_USER} = $DB_DSN[$i]->{user}; $ENV{ORA2PG_PASSWD} = $DB_DSN[$i]->{pwd}; # Used to pass additional information to ora2pg command my $info = ''; # Set RDBMS type $info = ' -m' if ($DB_DSN[$i]->{type} eq 'MYSQL'); $info = ' -M' if ($DB_DSN[$i]->{type} eq 'MSSQL'); # Add custom configuration file if set $info .= ' -c ' . $CONF_FILE if ($CONF_FILE); my $cmd_ora2pg = $ORA2PG_CMD . $info; my $audit = ''; $audit = " --audit_user \"$DB_DSN[$i]->{audit_user}\"" if ($DB_DSN[$i]->{audit_user}); # Extract SID or db name from the DSN # dbi:Oracle:host=foobar;sid=ORCL;port=1521 # dbi:Oracle:DB # dbi:Oracle://192.168.1.10:1521/XE # DBI:mysql:database=$db;host=$host if ($DB_DSN[$i]->{dsn} =~ m/(?:sid|database|service_name)=([^;]+)/ || $DB_DSN[$i]->{dsn} =~ m/dbi:Oracle:([\w]+)$/ || $DB_DSN[$i]->{dsn} =~ m/dbi:Oracle:\/\/[^\/]+\/([\w]+)/ ) { $DB_DSN[$i]->{sid} = $1; } elsif (!$DB_DSN[$i]->{schema}) { print "WARNING: couldn't determine sid/database name for DSN ". $DB_DSN[$i]->{dsn} .", without explicit schema can not processed this entry. Skiping.\n"; next; } else { $DB_DSN[$i]->{sid} = 'schema'; } # Extract host if ($DB_DSN[$i]->{dsn} =~ m/(?:host|server)=([^;]+)/ || $DB_DSN[$i]->{dsn} =~ m/dbi:Oracle:\/\/([^\/]+)/) { $DB_DSN[$i]->{host} = $1; $DB_DSN[$i]->{host} =~ s/:\d$+//; $DB_DSN[$i]->{host} .= '_'; } # When no schema or database is set, let Ora2Pg autodetect the list of available schema if ($DB_DSN[$i]->{schema} eq '') { if ($DRYRUN) { print "Running: $cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DSN[$i]->{dsn}'\n"; print `$cmd_ora2pg -t SHOW_SCHEMA -s "$DB_DSN[$i]->{dsn}"`; print "For each schema returned the following commands will be executed:\n"; print " $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"\" >> $OUTDIR${SEP}dbs_scan.csv\n"; print " $cmd_ora2pg -t SHOW_REPORT --dump_as_$FORMAT --cost_unit_value $COST_UNIT --estimate_cost$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"\" >> \"$OUTDIR${SEP}$DB_DSN[$i]->{host}$DB_DSN[$i]->{sid}_-report.$FORMAT\"\n"; } else { my @schema_list = `$cmd_ora2pg -t SHOW_SCHEMA -s "$DB_DSN[$i]->{dsn}"`; foreach my $line (@schema_list) { my ($type, $schemaname) = split(/\s+/, $line); $DB_DSN[$i]->{schema} = $schemaname; # Escape some chars for file path use $schemaname = quotemeta($schemaname); print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"$DB_DSN[$i]->{schema}\" >> $OUTDIR${SEP}dbs_scan.csv\n"; `$cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s "$DB_DSN[$i]->{dsn}" -n "$DB_DSN[$i]->{schema}" >> $OUTDIR${SEP}dbs_scan.csv`; $header = ''; print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_$FORMAT --cost_unit_value $COST_UNIT --estimate_cost$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"$DB_DSN[$i]->{schema}\" >> \"$OUTDIR${SEP}$DB_DSN[$i]->{host}$DB_DSN[$i]->{sid}_$schemaname-report.$FORMAT\"\n"; `$cmd_ora2pg -t SHOW_REPORT --dump_as_$FORMAT --cost_unit_value $COST_UNIT --estimate_cost$audit -s "$DB_DSN[$i]->{dsn}" -n "$DB_DSN[$i]->{schema}" >> "$OUTDIR${SEP}$DB_DSN[$i]->{host}$DB_DSN[$i]->{sid}_$schemaname-report.$FORMAT"`; } } } else { # Escape some chars for file path use my $schemaname = quotemeta($DB_DSN[$i]->{schema}); if ($DRYRUN) { print "Running: $cmd_ora2pg -t SHOW_SCHEMA -s \"$DB_DSN[$i]->{dsn}\" -n \"$DB_DSN[$i]->{schema}\"\n"; print `$cmd_ora2pg -t SHOW_SCHEMA -s "$DB_DSN[$i]->{dsn}" -n "$DB_DSN[$i]->{schema}"` } else { print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"$DB_DSN[$i]->{schema}\" >> $OUTDIR${SEP}dbs_scan.csv\n"; `$cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s "$DB_DSN[$i]->{dsn}" -n "$DB_DSN[$i]->{schema}" >> $OUTDIR${SEP}dbs_scan.csv`; print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_$FORMAT --cost_unit_value $COST_UNIT --estimate_cost$audit -s \"$DB_DSN[$i]->{dsn}\" -n \"$DB_DSN[$i]->{schema}\" >> \"$OUTDIR${SEP}$DB_DSN[$i]->{sid}_$schemaname-report.$FORMAT\"\n"; `$cmd_ora2pg -t SHOW_REPORT --dump_as_$FORMAT --cost_unit_value $COST_UNIT --estimate_cost$audit -s "$DB_DSN[$i]->{dsn}" -n "$DB_DSN[$i]->{schema}" >> "$OUTDIR${SEP}$DB_DSN[$i]->{sid}_$schemaname-report.$FORMAT"`; } } } exit 0; sub usage { my $msg = shift; print "$msg\n" if ($msg); print qq{ Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] -b | --binpath DIR: full path to directory where the ora2pg binary stays. Might be useful only on Windows OS. -c | --config FILE: set custom configuration file to use otherwise ora2pg will use the default: /etc/ora2pg/ora2pg.conf. -f | --format FMT: set the output format for the reports. Can be html or json. Default to html. -l | --list FILE : CSV file containing a list of databases to scan with all required information. The first line of the file can contain the following header that describes the format that must be used: "type","schema/database","dsn","user","password" -o | --outdir DIR : (optional) by default all reports will be dumped to a directory named 'output', it will be created automatically. If you want to change the name of this directory, set the name at second argument. -t | --test : just try all connections by retrieving the required schema or database name. Useful to validate your CSV list file. -u | --unit MIN : redefine globally the migration cost unit value in minutes. Default is taken from the ora2pg.conf (default 5 minutes). Here is a full example of a CSV databases list file: "type","schema/database","dsn","user","password" "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","HR","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","system","manager" The CSV field separator must be a comma. Note that if you want to scan all schemas from an Oracle instance you just have to leave the schema field empty, Ora2Pg will automatically detect all available schemas and generate a report for each one. Of course you need to use a connection user with enough privileges to be able to scan all schemas. For example: "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" "MSSQL","","dbi:ODBC:driver=msodbcsql18;server=srv.database.windows.net;database=testdb","system","manager" will generate a report for all schema in the XE instance. Note that in this case the SCHEMA directive in ora2pg.conf must not be set. }; exit 1; }