Oracle - OMS responded illegally [ERROR- Failed to Update Target Type Metadata]
We want to upgrade the JDK for EM agent.
Following the DOC:
How to Use JDK 7 Update 191 with EM 13.2 / 13.3 OMS (Doc ID 2241358.1)
The list of JDK:
All Java SE Downloads on MOS (Doc ID 1439822.1)
In order to upgrade JDK, one-off patch 28373706,28033677 should be applied on AGENT home in advance.
- 28373706 EM-AGENT Bundle Patch 13.2.0.0.180831
- 28033677 RTI:21262694 - jdk , Existing timezone are not considered as valid time zone
After applying 28373706 EM-AGENT Bundle Patch 13.2.0.0.180831, the agent cannot connect to OMS
$ /l01/app/oracle/product/oem/agent12c/agent_13.2.0.0.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.2.0.0.0
==> OMS Version : (unknown)
Protocol Version : 12.1.0.1.0
Agent Home : /l01/app/oracle/product/oem/agent12c/agent_inst
Agent Log Directory : /l01/app/oracle/product/oem/agent12c/agent_inst/sysman/log
Agent Binaries : /l01/app/oracle/product/oem/agent12c/agent_13.2.0.0.0
Core JAR Location : /l01/app/oracle/product/oem/agent12c/agent_13.2.0.0.0/jlib
Agent Process ID : 6356
Parent Process ID : 6319
Agent URL : https://tara.pprd.odu.edu:3872/emd/main/
Local Agent URL in NAT : https://tara.pprd.odu.edu:3872/emd/main/
Repository URL : https://ariel.prod.odu.edu:1159/empbs/upload
Started at : 2018-12-12 13:22:11
Started by user : oracle
Operating System : Linux version 2.6.32-754.3.5.el6.x86_64 (amd64)
Number of Targets : 18
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 38
Size of XML files pending upload(MB) : 0.04
Available disk space on upload filesystem : 51.99%
Collection Status : Collections enabled
==> Heartbeat Status : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]
Last attempted heartbeat to OMS : 2018-12-12 13:22:19
Last successful heartbeat to OMS : (none)
Next scheduled heartbeat to OMS : 2018-12-12 13:22:49
---------------------------------------------------------------
$ /l01/app/oracle/product/oem/agent12c/agent_13.2.0.0.0/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
==> EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)
$ tail -n 10 /l01/app/oracle/product/oem/agent12c/agent_inst/sysman/log/gcagent.log|grep WARN
2018-12-12 13:26:51,303 [176:9FC136A7] WARN - Ping protocol error
2018-12-12 13:27:19,514 [169:E866DA04:GC.SysExecutor.2 (AgentSystemMonitorTask)] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2018-12-12 13:27:19,514 [169:E866DA04] WARN - Subsystem (Ping Manager) returned bad status of {+ Ping Manager: *Warning* +}
2018-12-12 13:27:21,371 [182:BE87CC23] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: ERROR- Failed to Update Target Type Metadata)
2018-12-12 13:27:21,371 [182:BE87CC23] WARN - Ping protocol error
==> 2018-12-12 13:27:25,953 [43:BF500E2C] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: ERROR- Failed to Update Target Type Metadata)
This is because the mandatory patch 25163555 is not applied on OMS home in advance.
Agent Upload Error: OMS responded illegally [ERROR- Failed to Update Target Type Metadata] (Doc ID 2243040.1)
SQL> select type_meta_ver from sysman.mgmt_target_type_versions where target_type = 'oracle_emd'; TYPE_META_VER -------------------------------------------------------------------------------- 12.0 12.2 12.3 12.4 13.1 -- 13.2 should be here $ export ORACLE_HOME=/l01/app/oracle/product/oem/em13c $ $ORACLE_HOME/OPatch/opatch lsinventory|grep 25163555 -- 25163555 should be shown here
If 25163555 is not listed, we should rollback 28373706 on the agent and apply 25163555 on OMS
Patch 25163555: Tracking bug for Back-porting 24588124 oms side fix Release: Enterprise Manager Base Platform - OMS 13.2.0.0.0 Platform: Generic Platform Language: American English
After 25163555 is applied on OMS home, run the some query above to verify that 13.2 listed Then, patch agent again.
So, In summay, the following patch should be applied on OMSs or Agents in advance before upgrade JDK which Agents is using.
OMS
Apply 25163555 for OMS, then do testing.
Patch 25163555: Tracking bug for Back-porting 24588124 oms side fix
27612395 and 27994582 are prerequisites
Patch 27612395: EMBP Patch Set Update 13.2.0.0.180417
Patch 27994582: [EM 13.2]: April Agent BP deploy from Console fails with 13.20811 metadata error
Using 28970534 to instead.
# EM Cloud Control PSU : List of Available PSUs (Patch Set Updates) (Doc ID 1605609.1)
Patch 28970534: EMBP Patch Set Update 13.2.0.0.190115
AGENT
Patch 28966303: EM-AGENT Bundle Patch 13.2.0.0.181231
Patch 28033677: RTI:21262694 - jdk , Existing timezone are not considered as valid time zone
Patch 27895402: Oracle JDK 7 Update 191
$AGENT_INST_HOME/bin>./emctl stop agent Go to AGENT_HOME/oracle_common/ $mv jdk jdk_bak # unzip new jdk here $AGENT_INST_HOME/bin>./emctl start agent
Tomcat - Disable HTTP method
The way is very simple, if you want to disable HTTP methods for all applications on this Tomcat Add the following at conf/web.xml
Our requirement is to disable DELETE and OPTIONS methods
<security-constraint>
<web-resource-collection>
<web-resource-name>restricted methods</web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>DELETE</http-method>
<http-method>OPTIONS</http-method>
</web-resource-collection>
<auth-constraint />
</security-constraint>
How to verify it is working?
curl -v -X OPTIONS https://xxxx.pprd.xxxx.edu:5678/homepage.htm
For OPTIONS:
The left side, port 5678, DELETE is disabled on this Tomcat, we received a message that:
HTTP Status 403 – Forbidden
The right side, port 8441, DELETE is not disabled, all allowed method will be listed there:
Allow: GET, HEAD, POST, PUT, DELETE, OPTIONS
For DELETE:
The left side, port 5678, DELETE is disabled on this Tomcat, we received a message that:
Access to the requested resource has been denied
The right side, port 8441, DELETE is not disabled
The server understood the request but refuses to authorize it.
Oracle - How to strengthen Oracle Cloud Control Security?
The following Document you may need to read to do this:
Self-signed certificate & Untrusted CA
EM 13c, 12c: How to Configure the Enterprise Manager Cloud Control Management Agent for Secure Socket Layer (SSL) Certificates (Doc ID 2213661.1)
Disable TLSv1.0 / TLSv1.1 / SSLv3
EM 12c can just support TLS v1.0
EM 13c: Enterprise Manager 13c Cloud Control Configuration with Specific Transport Layer Security Protocol:TLSv1.0,TLSv1.1,TLSv1.2 (Doc ID 2212006.1)
EM 12c: Configure Enterprise Manager 12c Cloud Control to Accept Connections with TLSv1.0 Protocol (Doc ID 1602983.1)
Disable Weak Cipher Suites
13c: How to Disable Weak SSLCipherSuites in Enterprise Manager 13c Cloud Control (Doc ID 2138391.1)
EM 12c: How to Disable Weak SSLCipherSuites Used by Enterprise Manager 12c Cloud Control (Doc ID 1477287.1)
How To Disable Anonymous and Weak Cipher Suites in Oracle WebLogic Server (Doc ID 1067411.1)
If you are using EM13.2, JDK should be upgraded in advance to at least 1.7_131
How to Use JDK 7 Update 191 with EM 13.2 / 13.3 OMS (Doc ID 2241358.1)
Somethings you may need:
1-Determine the port of OMS / agent
$ /u01/app/oracle/agent/agent_13.3.0.0.0/bin/emctl status agent|grep URL Agent URL : https://oem13:3872/emd/main/ Local Agent URL in NAT : https://oem13:3872/emd/main/ Repository URL : https://oem13:4903/empbs/upload $ emctl status oms -details Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : oem13 HTTP Console Port : 7788 HTTPS Console Port : 7803 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://oem13:7803/em Upload URL: https://oem13:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : oem13 Admin Server HTTPS Port: 7102 Admin Server is RUNNING Oracle Management Server Information Managed Server Instance Name: EMGC_OMS1 Oracle Management Server Instance Host: oem13 WebTier is Up Oracle Management Server is Up JVMD Engine is Up BI Publisher Server Information BI Publisher Managed Server Name: BIP BI Publisher Server is Up BI Publisher HTTP Managed Server Port : 9701 BI Publisher HTTPS Managed Server Port : 9803 BI Publisher HTTP OHS Port : 9788 BI Publisher HTTPS OHS Port : 9851 BI Publisher is locked. BI Publisher Server named ‘BIP’ running at URL: https://oem13:9851/xmlpserver/servlet/home BI Publisher Server Logs: /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/ BI Publisher Log : /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log $ cd /u01/app/oracle/ $ find . -name nodemanager.properties ./gc_inst/user_projects/domains/GCDomain/nodemanager/nodemanager.properties $ cat ./gc_inst/user_projects/domains/GCDomain/nodemanager/nodemanager.properties|grep ListenPort ListenPort=7403
2-Check TLSv1.0 / TLSv1.1 / SSLv3 / Weak Cipher
$ openssl s_client -connect localhost:3872 -ssl3 $ openssl s_client -connect localhost:3872 -tls1 $ openssl s_client -connect localhost:3872 -tls1_1 $ openssl s_client -connect localhost:3872 -cipher MEDIUM
3-Determine WLS Version
$ source ./setWLSEnv.sh $ java weblogic.version
4-Change the password of SYSMAN in database
https://docs.oracle.com/cd/E73210_01/EMSEC/GUID-5DD3B11A-1159-40BD-8AEB-41EDE664AB12.htm#EMSEC12918
During password change, if see error
$ /u01/app/oracle/middleware/bin/emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd oracle -new_pwd Oracle$123 Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Changing passwords in backend … Passwords changed in backend successfully. Updating repository password in Credential Store… javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException Error occurred. Check the log /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
And also find exception in the log file
./gc_inst/em/EMGC_OMS1/sysman/log/secure.log
Refer the following document to deal with.
OPSS: Applying EM/OMS 12.1.0.3.3 PSU Patch 18604893 Fails. emctl.log Shows: Failed to find OMS details: JPS-01030: Cannot get credential. | JPS-10000: There was an internal error in the policy store.. (Doc ID 2015186.1)
5-Reset agent registration password
Enterprise Manager Cloud Control Security: How to Create or Edit the Agent Registration Password for Agent to OMS Secure Communication (Doc ID 1367946.1)
Oracle - ORION
Orion is a disk performance test tool which is provided by Oracle.
It used to simulate the characteristic of different type of database IO, to do pressure test on the disk.
For example:
- Small random IO: index range scan, access row from index.
- Large random IO: due to the disk striping, a sequential read is extended to more than one disks, so, on the disk level, the continuous read can be seem as random IO with size 1MB.
- Large Sequential IO: data loading / backup / restore. Also for parallel query which subprocess can coordinate together.
- mixed load.
Orion supports different type of files on the storage.
For example:
- raw disks, and simulate striping, as ASM.
- raw disks, don’t simulate striping, as used for datafile directly.
- filesystem, create files on it and use files for testing, don’t simulate striping
The options of Orion
-run
- simple: Tests random small (8K) IOs at various loads, show IOPS & Latency, then random large (1M) IOs at various loads, show Mbps
- normal: Tests combinations of random small (8K) IOs and random large (1M) IOs. Do testing for the combination of n*m matrix, for all of three testing.
- advanced: Tests the workload specified by the user using optional parameters
- dss: Tests with random large (1M) IOs at increasing loads to determine the maximum throughput. Only show Mbps
- oltp: Tests with random small (8K) IOs at increasing loads to determine the maximum IOPS. Only show IOPS and Latency
-testname
- Name of the test run. Orion read the file named .lun which is a list of disks or files for testing.
-num_disks
- Number of disks (physical spindles). This number is used to gauge the range of loads that Orion should test at. Increasing this parameter results in Orion using heavier IO loads. Default is the number of LUNs in .lun.
-size_small
- Size of small IOs in KB. Default is 8. Normally, it should be set to the same as db_block_size of the database you want to create.
-size_large
- Size of large IOs in KB. Default is 1024. It should be set to the maximum IO size of OS. Normally, it is 1MB for the most of OS
-type
- Type of large IOs (rand, seq). Default is rand.
- The full table scan is Randomly distributed large IO
- And the data loading / backup / restore and parallel query are Sequential streams of large IOs
-num_streamIO
- Number of concurrent IOs per stream. This parameter is only used if -type is seq. Default is 4.
- It is equal to a query in Oracle database with /+parallel(a,4)/
-simulate
- Orion tests on a virtual LUN formed by combining the specified LUNs in one of these ways. This parameter is typically only used if -type is seq. Default is raid0.
- concat - A serial concatenation of the LUNs. Each sequential stream issues IOs to only one LUN.
- raid0 - A RAID-0 mapping across the LUNs. Each sequential stream issues IOs across all LUNs, using RAID-0 striping.
- If you are using ASM, regardless whether you do striping in storage, the ASM will do striping again on the ASM level, so, it should be set to raid0.
- If you are using raw devices as datafiles, it should be set to concat. Orion does testing on each file.
-write
- Percentage of IOs that are writes (SEE WARNING ABOVE). Default is 0.
- For read testing, set to 0, for write testing, set to 100, for mixed, set to 1-99
-cache_size
- Size in MBs of the array’s cache. Unless this option is set to 0, Orion issues a number of unmeasured, random IOs before each large sequential data point.
- These IOs fill up the storage array’s cache (if any) with random data so that IOs from one data point do not result in cache hits for the next data point. Reading tests are preceded with junk reads and writing tests are preceded with junk writes.
- If specified, this ‘cache warming’ is performed until cache_size MBs of IO have been read or written. Default behavior is to issue 2 minutes of unmeasured random IOs before each data point.
- This parameter is used to reduce the impact of the cache on the test. To large value consumes a lot of additional time for testing
-duration
- Duration of each data point in seconds. Default is 60.
-nosleep_on_delay
- Polls the timer instead of sleeping when a delay set between IO requests
-iorate_small
- Rate of small IOs that needs to be maintained for duration of test. Not used by default.
-iorate_large
- Rate of large IOs that needs to be maintained for duration of test. Not enabled by default
-num_small
- Number of outstanding small IOs. This parameter controls the small IO load. Only used if -matrix is point, col, or max. No default.
-num_large
- This parameter controls the large IO load. For -type rand, number of outstanding large IOs. For -type seq, number of sequential IO streams. Only used if -matrix is point, row, or max. No default.
-matrix
- An Orion test consists of multiple data point tests. These data point tests can be represented as a two-dimensional matrix. Each column in the matrix represents data point tests with the same small IO load, but varying large IO loads. Each row represents data point tests with the same large IO load, but varying small IO loads. An Orion test can be for a single point, a single row, a single column, or the whole matrix, depending on the matrix option setting below. Default is basic.
- basic - Test small IOs only, then large IOs only.
- detailed - Test entire matrix.
- point - Test with num_small small IOs, num_large large IOs.
- col - Test a varying large IO load with num_small small IOs.
- row - Test a varying small IO load with num_large large IOs.
- max - Test varying loads up to the num_small and num_large limits.
-hugenotneeded
- Tells Orion to allocate regular pages when huge pages are not available. With out this option, on systems that support hugepages, Orion errors out when it cannot allocate sufficient huge pages. -hugenotneeded is not set by default.
- If huge page is not configured yet, use this parameter.
-verbose
- Prints tracing information to standard output if set. Not set by default.
-large_io_hist
- Prints detailed output with Large IO latency histogram and other detailed information for large and small IOs. Not enabled by default.
- Latency information will be reported in the summary and hist, but the chart of IOPS and latency is still empty.
-datainput
- Name of the input file for data to be written.
-is_lgwr
- If set to true will run test using VIP level QOS, assuming the underlying transport supports it.
- VIP: Versatile Interface Processors
- QOS: Quality of Service
-is_rdma_lgwr
- If set to true will run test using VIP level and rdma hint assuming the underlying transport supports it.
- rdma: Remote direct memory access
-no_hi_bw
- If present won’t run as high-bandwidth client
-iotype
- IO Modes: async, directio, setall, none. (Default is setall)
- If using filesystem for database, set it as filesystemio_options in database.
-rtstats_interval
- Interval (in seconds) at which the cumulative statistics of the last interval is printed on stdout
-hist_granule
- Granularity (in usecs) of latency histogram
-filetype
- Access filetype to use. Consult skgfr.h:skgfrftnm for valid values.
-writereadonce
- Set the IORM Read/Write Once flag.
Normally, we can do the following tests:
large IO read
/u01/app/18.3.0/grid/bin/orion -run dss -testname dbtest -large_io_hist -hugenotneeded -verbose
small IO read
/u01/app/18.3.0/grid/bin/orion -run oltp -testname dbtest -hugenotneeded -verbose
Parallel 32 read
/u01/app/18.3.0/grid/bin/orion -run advanced -testname dbtest -write 0 -matrix max -num_large 1 -num_small 0 -type seq -num_streamIO 32 -hugenotneeded -verbose
DBWR write, 8 processes
/u01/app/18.3.0/grid/bin/orion -run advanced -testname dbtest -size_small 8 -write 100 -matrix max -num_large 0 -num_small 8 -hugenotneeded -verbose
Data loading, parallel 32
/u01/app/18.3.0/grid/bin/orion -run advanced -testname dbtest -size_small 32 -write 100 -matrix max -num_large 1 -num_small 0 -type seq -num_streamIO 32 -hugenotneeded -verbose
/u01/app/18.3.0/grid/bin/orion -run advanced -testname dbtest -size_small 8 -write 100 -matrix max -num_large 8 -num_small 0 -hugenotneeded -verbose
Mixed
/u01/app/18.3.0/grid/bin/orion -run normal -testname dbtest -hugenotneeded -verbose
The following options we can use to adjust the testing
to reduce the time for testing
-duration 10
to ignore the impact of cache and save testing time
-cache_size 0
if raw devices are used as datafiles directly
-simulate concat
if the db_block_size is not 8K
-size_small 32
if the IO maximum size is not equal to 1M
-size_large 2048
Oracle - NETPERF
NETPERF is an open-source and free tool to do a pressure test for the network, especially for interconnect networks of RAC. Normally, we need to do this before we install a new RAC.
We can get it from Github: https://github.com/HewlettPackard/netperf
Compile & Install:
./autogen.sh ./configure make make install
Before we do testing, we should start netserver on another side, for here, it is running on the box database02
[root@database02 netperf]# netserver [root@database02 netperf]# ps -ef|grep netserver root 20948 1 0 15:29 ? 00:00:00 netserver root 21871 20308 0 16:46 pts/0 00:00:00 grep --color=auto netserver
By then, we do test on the box database01
By default, netperf do the test of TCP throughput, it is the same with -t TCP_STREAM
[root@database01 ~]# netperf -H 83.16.16.42 -l 10 -f M -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 83.16.16.42 () port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB 87380 16384 16384 10.03 99.21 10.92 40.86 4.299 16.089
UDP throughput testing
[root@database01 ~]# netperf -H 83.16.16.42 -l 10 -f M -c -C -t UDP_STREAM MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 83.16.16.42 () port 0 AF_INET Socket Message Elapsed Messages CPU Service Size Size Time Okay Errors Throughput Util Demand bytes bytes secs # # MBytes/sec % SS us/KB 262144 65507 10.00 11298 0 70.6 45.13 24.978 262144 10.00 11283 70.5 14.36 7.949
TCP Request / Response Test
[root@database01 ~]# netperf -H 83.16.16.42 -l 10 -c -C -t TCP_RR MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 83.16.16.42 () port 0 AF_INET : first burst 0 Local /Remote Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem Send Recv Size Size Time Rate local remote local remote bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr 16384 87380 1 1 10.00 1655.86 7.92 10.31 191.222 249.143 16384 87380
UDP Request / Response Test
[root@database01 ~]# netperf -H 83.16.16.42 -l 10 -c -C -t TCP_RR MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 83.16.16.42 () port 0 AF_INET : first burst 0 Local /Remote Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem Send Recv Size Size Time Rate local remote local remote bytes bytes bytes bytes secs. per sec % S % S us/Tr us/Tr 16384 87380 1 1 10.00 1655.86 7.92 10.31 191.222 249.143 16384 87380
When we do testing, we can use ifstat for monitoring
[root@database01 ~]# watch -n 1 ifstat net0
Every 10.0s: ifstat net0 Mon Dec 3 16:15:50 2018
#kernel
Interface RX Pkts/Rate TX Pkts/Rate RX Data/Rate TX Data/Rate
RX Errs/Drop TX Errs/Drop RX Over/Rate TX Coll/Rate
net0 682146 0 20640 0 103M 0 136K 0
0 0 0 0 0 0 0 0
