class Builder extends AnyRef
Builder for KinesisInputDStream instances.
- Annotations
- @Evolving()
- Since
2.2.0
- Alphabetic
- By Inheritance
- Builder
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new Builder()
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
build(): KinesisInputDStream[Array[Byte]]
Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].
Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].
- returns
Instance of KinesisInputDStream constructed with configured parameters
-
def
buildWithMessageHandler[T](handler: (Record) ⇒ T)(implicit arg0: ClassTag[T]): KinesisInputDStream[T]
Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.
Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.
- handler
Function converting Record instances read by the KCL to DStream type T
- returns
Instance of KinesisInputDStream constructed with configured parameters
-
def
checkpointAppName(appName: String): Builder
Sets the KCL application name to use when checkpointing state to DynamoDB.
Sets the KCL application name to use when checkpointing state to DynamoDB. This is a required parameter.
- appName
Value to use for the KCL app name (used when creating the DynamoDB checkpoint table and when writing metrics to CloudWatch)
- returns
Reference to this KinesisInputDStream.Builder
-
def
checkpointInterval(interval: Duration): Builder
Sets how often the KCL application state is checkpointed to DynamoDB.
Sets how often the KCL application state is checkpointed to DynamoDB. Defaults to the Spark Streaming batch interval if no custom value is specified.
- interval
Duration specifying how often the KCL state should be checkpointed to DynamoDB.
- returns
Reference to this KinesisInputDStream.Builder
-
def
clone(): AnyRef
- Attributes
- protected[java.lang]
- Definition Classes
- AnyRef
- Annotations
- @native() @throws( ... )
-
def
cloudWatchCredentials(credentials: SparkAWSCredentials): Builder
Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint.
Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.
- credentials
SparkAWSCredentials to use for CloudWatch authentication
-
def
dynamoDBCredentials(credentials: SparkAWSCredentials): Builder
Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint.
Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.
- credentials
SparkAWSCredentials to use for DynamoDB authentication
-
def
endpointUrl(url: String): Builder
Sets the AWS Kinesis endpoint URL.
Sets the AWS Kinesis endpoint URL. Defaults to "https://kinesis.us-east-1.amazonaws.com" if no custom value is specified
- url
Kinesis endpoint URL to use
- returns
Reference to this KinesisInputDStream.Builder
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[java.lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initialPosition(initialPosition: KinesisInitialPosition): Builder
Sets the initial position data is read from in the Kinesis stream.
Sets the initial position data is read from in the Kinesis stream. Defaults to KinesisInitialPositions.Latest if no custom value is specified.
- initialPosition
KinesisInitialPosition value specifying where Spark Streaming will start reading records in the Kinesis stream from
- returns
Reference to this KinesisInputDStream.Builder
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
kinesisCredentials(credentials: SparkAWSCredentials): Builder
Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint.
Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint. Defaults to DefaultCredentialsProvider if no custom value is specified.
- credentials
SparkAWSCredentials to use for Kinesis authentication
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
regionName(regionName: String): Builder
Sets the AWS region to construct clients for.
Sets the AWS region to construct clients for. Defaults to "us-east-1" if no custom value is specified.
- regionName
Name of AWS region to use (e.g. "us-west-2")
- returns
Reference to this KinesisInputDStream.Builder
-
def
storageLevel(storageLevel: StorageLevel): Builder
Sets the storage level of the blocks for the DStream created.
Sets the storage level of the blocks for the DStream created. Defaults to StorageLevel.MEMORY_AND_DISK_2 if no custom value is specified.
- storageLevel
StorageLevel to use for the DStream data blocks
- returns
Reference to this KinesisInputDStream.Builder
-
def
streamName(streamName: String): Builder
Sets the name of the Kinesis stream that the DStream will read from.
Sets the name of the Kinesis stream that the DStream will read from. This is a required parameter.
- streamName
Name of Kinesis stream that the DStream will read from
- returns
Reference to this KinesisInputDStream.Builder
-
def
streamingContext(jssc: JavaStreamingContext): Builder
Sets the StreamingContext that will be used to construct the Kinesis DStream.
Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.
- jssc
JavaStreamingContext used to construct Kinesis DStreams
- returns
Reference to this KinesisInputDStream.Builder
-
def
streamingContext(ssc: StreamingContext): Builder
Sets the StreamingContext that will be used to construct the Kinesis DStream.
Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.
- ssc
StreamingContext used to construct Kinesis DStreams
- returns
Reference to this KinesisInputDStream.Builder
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @throws( ... )
Deprecated Value Members
-
def
initialPositionInStream(initialPosition: InitialPositionInStream): Builder
Sets the initial position data is read from in the Kinesis stream.
Sets the initial position data is read from in the Kinesis stream. Defaults to InitialPositionInStream.LATEST if no custom value is specified. This function would be removed when we deprecate the KinesisUtils.
- initialPosition
InitialPositionInStream value specifying where Spark Streaming will start reading records in the Kinesis stream from
- returns
Reference to this KinesisInputDStream.Builder
- Annotations
- @deprecated
- Deprecated
(Since version 2.3.0) use initialPosition(initialPosition: KinesisInitialPosition)