Performance Issue since Solr 7.7 with wt=javabin
コピーリンク
Hi Solr team,
we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.
Just an example: Doing a simple query for . and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!
The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.
To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:
Solr 7.6:
docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)
Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)
For me it seems like a bug. But if not, then please let me know what I did wrong ;-)
Best Regards,
Andy Reek
Principal Software Developer
diva-e Jena
Mälzerstraße 3, 07745 Jena, Deutschland
T: +49 (3641) 3678 (223)
F: +49 (3641) 3678 101
andy.reek@diva-e.comandy.reek@diva-e.com
www.diva-e.comhttp://www.diva-e.com follow us: facebookhttps://www.facebook.com/digital.value.enterprise/?ref=hl, twitterhttps://twitter.com/diva_enterprise, LinkedInhttps://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh, Xinghttps://www.xing.com/companies/diva-edigitalvalueenterprisegmbh
diva-e AGETO GmbH
Handelsregister: HRB 210399 Amtsgericht Jena
Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
添付ファイル：
Hi Solr team,
we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.
Just an example: Doing a simple query for . and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!
The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.
To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:
Solr 7.6:
docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)
Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)
For me it seems like a bug. But if not, then please let me know what I did wrong ;-)
Best Regards,
Andy Reek
Principal Software Developer
diva-e Jena
Mälzerstraße 3, 07745 Jena, Deutschland
T: +49 (3641) 3678 (223)
F: +49 (3641) 3678 101
andy.reek@diva-e.comandy.reek@diva-e.com
www.diva-e.comhttp://www.diva-e.com follow us: facebookhttps://www.facebook.com/digital.value.enterprise/?ref=hl, twitterhttps://twitter.com/diva_enterprise, LinkedInhttps://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh, Xinghttps://www.xing.com/companies/diva-edigitalvalueenterprisegmbh
diva-e AGETO GmbH
Handelsregister: HRB 210399 Amtsgericht Jena
Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
添付ファイル：
How are you consuming the output? Are you using solrj?
On Tue, Jun 18, 2019, 1:27 AM Andy Reek Andy.Reek@diva-e.com wrote:
Hi Solr team,
we are using Solr in version 7.1 as search engine in our online shop (SAP
Hybris). And as a task I needed to migrate to the most recent Solr in
version 7 (7.7). Doing this I faced extreme performance issues. After
debugging and testing different setups I found out, that they were caused
by the parameter wt=javabin. These issues begin to raise since version 7.7,
in 7.6 it is still working as fast as in 7.1.Just an example: Doing a simple query for . and wt=javabin in 7.6: 0.2
seconds and in 7.7: 34 seconds!The configuration of the schema.xml and solrconfig.xml are equal in both
versions. Version 8.1 has the same effect as 7.7. Using something other
than wt=javabin (e.g. wt=xml) will work fast in every version - which is
our current workaround.To reproduce this issue I have attached my used configsets folder plus
some test data. This all can be tested with docker and wget:Solr 7.6:
docker run -d --name solr7.6 -p 8983:8983 --rm -v
$PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
solr:7.6-slim solr-create -c mycore -d
/opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v
$PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
solr:7.7-slim solr-create -c mycore -d
/opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)For me it seems like a bug. But if not, then please let me know what I did
wrong ;-)Best Regards,
Andy Reek
Principal Software Developer
diva-e Jena
Mälzerstraße 3, 07745 Jena, Deutschland
T: +49 (3641) 3678 (223)
F: +49 (3641) 3678 101
andy.reek@diva-e.com andy.reek@diva-e.com
www.diva-e.com http://www.diva-e.com follow us: facebook
https://www.facebook.com/digital.value.enterprise/?ref=hl, twitter
https://twitter.com/diva_enterprise, LinkedIn
https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh,
Xing https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh
diva-e AGETO GmbH
Handelsregister: HRB 210399 Amtsgericht Jena
Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
Hi,
Did you find a solution to your performance problem?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
- jun. 2019 kl. 17:17 skrev Andy Reek Andy.Reek@diva-e.com:
Hi Solr team,
we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.
Just an example: Doing a simple query for . and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!
The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.
To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:
Solr 7.6:
docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999 http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)For me it seems like a bug. But if not, then please let me know what I did wrong ;-)
Best Regards,
Andy Reek
Principal Software Developer
diva-e Jena
Mälzerstraße 3, 07745 Jena, DeutschlandT: +49 (3641) 3678 (223)
F: +49 (3641) 3678 101
andy.reek@diva-e.com andy.reek@diva-e.com
www.diva-e.com http://www.diva-e.com/ follow us: facebook https://www.facebook.com/digital.value.enterprise/?ref=hl, twitter https://twitter.com/diva_enterprise, LinkedIn https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh, Xing https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh
diva-e AGETO GmbH
Handelsregister: HRB 210399 Amtsgericht Jena
Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
Hi Andy,
Have you run performance benchmarking for sometime and made sure that the
Solr Caching and GC doesn't impact the performance? I recommend that you
should rebuild the performance matrix after few warmups and requests. Have
you invalidated this?
On Fri, 18 Oct 2019 at 12:35, Jan Høydahl jan.asf@cominvent.com wrote:
Hi,
Did you find a solution to your performance problem?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
- jun. 2019 kl. 17:17 skrev Andy Reek Andy.Reek@diva-e.com:
Hi Solr team,
we are using Solr in version 7.1 as search engine in our online shop
(SAP Hybris). And as a task I needed to migrate to the most recent Solr in
version 7 (7.7). Doing this I faced extreme performance issues. After
debugging and testing different setups I found out, that they were caused
by the parameter wt=javabin. These issues begin to raise since version 7.7,
in 7.6 it is still working as fast as in 7.1.Just an example: Doing a simple query for . and wt=javabin in 7.6: 0.2
seconds and in 7.7: 34 seconds!The configuration of the schema.xml and solrconfig.xml are equal in both
versions. Version 8.1 has the same effect as 7.7. Using something other
than wt=javabin (e.g. wt=xml) will work fast in every version - which is
our current workaround.To reproduce this issue I have attached my used configsets folder plus
some test data. This all can be tested with docker and wget:Solr 7.6:
docker run -d --name solr7.6 -p 8983:8983 --rm -v
$PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
solr:7.6-slim solr-create -c mycore -d
/opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "
http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999 http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v
$PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
solr:7.7-slim solr-create -c mycore -d
/opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)For me it seems like a bug. But if not, then please let me know what I
did wrong ;-)Best Regards,
Andy Reek
Principal Software Developerdiva-e Jena
Mälzerstraße 3, 07745 Jena, DeutschlandT: +49 (3641) 3678 (223)
F: +49 (3641) 3678 101
andy.reek@diva-e.com andy.reek@diva-e.comwww.diva-e.com http://www.diva-e.com/ follow us: facebook https://www.facebook.com/digital.value.enterprise/?ref=hl, twitter https://twitter.com/diva_enterprise, LinkedIn https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh,
Xing https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh
diva-e AGETO GmbH
Handelsregister: HRB 210399 Amtsgericht Jena
Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
--
Regards,
Paras Lehana [65871]
Software Programmer, Auto-Suggest,
IndiaMART Intermesh Ltd.
8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303
Mob.: +91-9560911996
Work: 01203916600 | Extn: 8173
--
IMPORTANT:
NEVER share your IndiaMART OTP/ Password with anyone.
Hi Paras,
Thanks for your answer and your ideas ;)
I have the exact same issue than Andy "wt=javabin&version=2" have really
poor performances comprared to wt=json
I'm using :
- solr 7.7.2
- OpenJDK8U-jdk_x64_linux_hotspot_8u222b10 or jdk-8u241-linux-x64 (same
behaviour)
The server have much RAM and GC are not triggered during the test.
I alternatively perform stress test with wt=javabin then wt=json without
restarting solr. I presume warmups is not an issue there.
What do you mean by "rebuild the performance matrix"
Thanks
Florent
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Check out SOLR-14013 https://issues.apache.org/jira/browse/SOLR-14013 which I believe is what you are looking for
Jan
- jan. 2020 kl. 11:46 skrev Florent Sithi florent.public@gmail.com:
Hi Paras,
Thanks for your answer and your ideas ;)
I have the exact same issue than Andy "wt=javabin&version=2" have really
poor performances comprared to wt=json
I'm using :
- solr 7.7.2
- OpenJDK8U-jdk_x64_linux_hotspot_8u222b10 or jdk-8u241-linux-x64 (same
behaviour)The server have much RAM and GC are not triggered during the test.
I alternatively perform stress test with wt=javabin then wt=json without
restarting solr. I presume warmups is not an issue there.What do you mean by "rebuild the performance matrix"
Thanks
Florent--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
yes thanks so much, fixed in 8.4.0
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Could anyone produce a patch for 7.7 please?
From: Florent Sithi florent.public@gmail.com
Sent: 29 January 2020 14:34
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabin
yes thanks so much, fixed in 8.4.0
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
No further releases are planned for 7.x, so your best bet is to patch branch_7_7 yourself and build a custom Solr version.
Jan
- jan. 2020 kl. 20:54 skrev Karl Stoney Karl.Stoney@autotrader.co.uk.INVALID:
Could anyone produce a patch for 7.7 please?
From: Florent Sithi florent.public@gmail.com
Sent: 29 January 2020 14:34
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabinyes thanks so much, fixed in 8.4.0
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
I don’t have confidence in my ability to do that, I was hoping someone could help out as moving to 8.4 is too much of a jump for me right now!
Would really appreciate it..
Get Outlook for iOShttps://aka.ms/o0ukef
From: Jan Høydahl jan.asf@cominvent.com
Sent: Thursday, January 30, 2020 2:23:40 PM
To: solr-user solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabin
No further releases are planned for 7.x, so your best bet is to patch branch_7_7 yourself and build a custom Solr version.
Jan
- jan. 2020 kl. 20:54 skrev Karl Stoney Karl.Stoney@autotrader.co.uk.INVALID:
Could anyone produce a patch for 7.7 please?
From: Florent Sithi florent.public@gmail.com
Sent: 29 January 2020 14:34
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabinyes thanks so much, fixed in 8.4.0
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
To be specific sorry, we already build 77 from source, I don’t have confidence in back porting the fix however so it would be awesome if someone would help out other 77 users who aren’t able to upgrade to 84 yet with this important fix :(
Get Outlook for iOShttps://aka.ms/o0ukef
From: Karl Stoney Karl.Stoney@autotrader.co.uk.INVALID
Sent: Thursday, January 30, 2020 3:56:31 PM
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabin
I don’t have confidence in my ability to do that, I was hoping someone could help out as moving to 8.4 is too much of a jump for me right now!
Would really appreciate it..
From: Jan Høydahl jan.asf@cominvent.com
Sent: Thursday, January 30, 2020 2:23:40 PM
To: solr-user solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabin
No further releases are planned for 7.x, so your best bet is to patch branch_7_7 yourself and build a custom Solr version.
Jan
- jan. 2020 kl. 20:54 skrev Karl Stoney Karl.Stoney@autotrader.co.uk.INVALID:
Could anyone produce a patch for 7.7 please?
From: Florent Sithi florent.public@gmail.com
Sent: 29 January 2020 14:34
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Subject: Re: Performance Issue since Solr 7.7 with wt=javabinyes thanks so much, fixed in 8.4.0
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
トピックへ返信するには、ログインが必要です。