112,841 questions
0
votes
0
answers
17
views
Data is writing in Redis using the format table:key (e.g., users:1001, users:1002), where each key is a separate hash in Spark
each row writing as a separate Redis hash with keys like "table:row_id" format. As shown in the attached screenshot, but we need to write keys in single hash
Is there any way to write data ...
- 49
1
vote
1
answer
33
views
Unable to read JSON from Redis Cache using spark
I am unable to read data from the Redis cache; although the keys are present, the query returns an empty DataFrame in Spark
redis_key
json
val tableDF = sqlContext.read.format("org.apache.spark....
- 49
0
votes
1
answer
47
views
Scala Spark decompress lz4 csv
Is there any way to decompress csv lz4 files by Spark?
I tried following approaches:
First:
sparkSession.read
.option("delimiter", ",")
.option("compression", "...
- 1,394
0
votes
2
answers
61
views
Read files from storage account in Spark - Without using keys - Azure
I am doing local development. Wish to run spark job locally on my desktop, and access files in storage account from my spark job.
I don't have an option to use SAS tokens or access-keys for my storage ...
- 31.8k
0
votes
1
answer
32
views
bigtable spark connector not reading cell timestamp with the data
I am using bigtable spark connector to read bigtable data in scala code. I want to read the cell timestamp with the data as well. But nowhere I can find how to do it. Can someone help me on this?
I ...
- 226
0
votes
1
answer
32
views
Global Temp view shows up empty when passed from Pyspark to Scala notebook in Databricks
I have a Azure Databricks workflow which runs a Pyspark notebook which in turns calls a Scala notebook(legacy) for a list of tables. In Pyspark notebook, I save a Dataframe to a GlobalTempView and ...
- 3
0
votes
0
answers
62
views
Unable to read schema from Karapace Schema registry using Confluent client, the curl command works fine
I'm unable to read schema from Karapace Schema registry using Confluent client, the curl command works fine
here is the scala code -
def getSchemaFromRegistry(schemaRegistryUrl: String, subject: ...
- 1,093
0
votes
0
answers
74
views
How to fail Spark on wrong schema?
The data:
{"name": "name1", "id": 1}
{"name": "name2", "id": "1"}
The code:
val schema =
"""
| name ...
- 33.9k
2
votes
0
answers
63
views
SyntacticRule with optional Semantic in Scalafix?
I am working on Rules to switch from Kyo 0.19 to Kyo 1.0, is there a way to make those rules Syntactic, with some optional Symbol checks if it's possible (semanticDb is available)?
package kyo.rules
...
- 1,769
0
votes
0
answers
89
views
Google Sign-In fails silently with /login?error when using HTTPS (works on HTTP) in Spring Boot + Scala application
I am facing an issue with the Google Sign-In integration in my Scala + Spring Boot web application.
Problem:
When I try to sign in with Google using HTTPS (deployed via ngrok), I am silently ...
0
votes
2
answers
60
views
sbt plugin resolvers not adding sbtVersion, scalaVersion to artifactory url
In an sbt project, I'm adding some plugins, like
addSbtPlugin("org.typelevel" % "sbt-tpolecat" % "0.5.2")
This should resolve to an artifact url similar to this:
https://...
- 573
0
votes
1
answer
40
views
Generic way of array creation in Scala.js
I'm looking for a generic way of array creation in Scala.js.
The following code works fine in JVM and Scala Native platforms:
def newArray[A](fqcn: String, size: Int): Array[A] = {
val clazz = Class....
- 8,033
1
vote
1
answer
59
views
Referring to non-existent class while compiling to ScalaJS
I'm builing a ScalaJs & Scala3 application, when i run the npm run dev/build command I have the following error, and I'm not figuring out how to fix it:
Referring to non-existent class scala....
0
votes
2
answers
68
views
conditional map fetch efficiently
I have a map which lays down prices of commodities in different currencies
val commpricemap: Map[String , Map[String, Double]] = ???
AN example of an entry for gold is as below:
("AU" -> ...
- 9,927
0
votes
0
answers
30
views
mill debugger intellij in Xiangshan project
I'm trying to debug the XiangShan RISC-V SoC project using IntelliJ IDEA with Mill and BSP.
I've followed all recommended steps (scala, the mill build tool, and the IntelliJ debugger) but breakpoints ...